Update: This is Part 1 in a series of personal projects investigating the use of a Social Robot (and in future other assistive technologies) to help manage Crohn’s Disease.
Please consider making a donation to the Irish Society for Colitis and Crohn’s Disease.
To anyone from Anki who might happen to read this: I was genuinely sad to hear about the company going out of business. I hope you all find work really soon and continue to make wonderful robots. Vector is an incredibly useful platform. I can’t express just how much it means to me. Your work helped me through a very very tough time and allowed me to give a giant middle finger to a disease that drastically changed the course of my life almost 13 years ago.
How It All Works
For those who aren’t interested in the technology: I have a scheduled program that checks a calendar for my daily medication every morning and runs various reminder and celebration animations that I’ve programmed on a social robot called “Vector“. If you want to skip ahead the videos of Vector in action start here.
Table of Contents
- Inspiration
- Raspberry Pi
- The Front End
- Flask
- Google Calendar
- Cron
- SQLite
- Using Vector
- The Videos
- Fiero
- Utterances (Text to Speech)
- Why not just use a smartphone?
- Future Work
Inspiration
The idea for this project came to me when I was dealing with a series of depressive “moods” after I began the loading dose of adalimumab. I can only describe them as an oppressive sadness, a physical weight in my chest, and they would come out of nowhere. I really had to fight to get a hold of my brain to be able to get through the day. They lasted for several months on and off. It wore me out but also really pissed me off and I wasn’t going to take it lying down.
I’ve been interested in “Social Robotics” since 2006 after reading a paper on Joint Attention while studying at the University of Sussex. I can’t remember who wrote the paper anymore. It was possibly Yukie Nagai, Cynthia Breazeal or Brian Scassellati given the time frame. I was immediately fascinated with just how difficult a problem it was to try to model and was really considering it as a thesis topic. There was a bit of a boom in the field at the time which was also called “Developmental Robotics” or “Epigenetic Robotics”.
I never got to finish the MSc because I became seriously ill. I would be eventually diagnosed with Crohn’s Disease. I had to postpone for a year after almost getting through the first semester and when I tried to go back I really wasn’t well enough to complete it. It was bitterly disappointing. I absolutely loved what I was doing and even had an EPSRC studentship to cover the fees. It took a while to accept that I really tried as hard as I could and a Postgraduate Diploma was still pretty bloody good under the circumstances.
For anyone unfamiliar with it Crohn’s Disease can feel like a full time job on top of a full time job while also trying to live a life outside of BOTH of those full time jobs. It’s a lot of medication taken on a schedule, a strict diet, physical pain, exhaustion, depression, anxiety, surgeries and many other issues. A particular nasty twist to the disease is that it is very individualised. While there are common symptoms there is no guarantee that a treatment plan that works for one patient will work for another. In fact within a patient’s body different areas of the disease can respond differently to medication even if the areas are located next to each other. The disease is always there in the background and (as has happened to me twice now) your body can stop responding to treatment that was working.
Fortuitously I regularly listen to the Cynical Developer podcast and it so happened that there was a three part series on Machine Learning last year (Part 1, Part 2, Part 3). The guest, Ed Yau, mentioned the Anki Vector.
I checked it out on YouTube and once I read that there would be an SDK I immediately knew what I wanted to do with it. Having missed their Kickstarter it was a stroke of good luck that Santa Claus got my email in time!
Raspberry Pi
I have a Raspberry Pi3 Model B+ with Python 3.6.1 and the Anki Vector SDK installed on it.
The Vector SDK allows me to connect to Vector over my home WiFi and run Python scripts that I’ve written that access functionality on the robot itself. Anki stated security reasons for not allowing people to write software that could be installed on the robots.
As I’m sure with everyone who builds a project on the weekends there’s always more you want to do with it. I had to be really strict with myself or I’d never get this finished and be able to actually use the thing. If you’re unfamiliar with it you should read about “Parkinson’s Law” which is often the nail in any hobby or side project coffin.
The Front End
The front end (front-end? frontend?) is a page built with JavaScript, HTML and CSS running on a Flask server on the Pi. It’s a responsive layout so I can run it off a laptop, a phone, iPad or even my TV’s browser (Samsung Browser 1.1 running on Tizen Linux if you’re curious). Why not use the power of the web? It calls code on the server side to query a Google Calendar for whatever medication I have to take for the current day and the next day (both stored as Calendar Events), displays the current amount of each medication I have (with warnings if I’m running low on any of them). It also allows me to update the levels of medication I have when I fill a new prescription (something I’d like to try to automate next).
I can mark the relevant medication as taken (click or tap on the item in Today’s list) and it will update my Google calendar to mark that item as “done”. An advantage of using an external Calendar that if someone else – a family member or care giver – wanted to know if I’d taken my medication or not they can see it through their shared access to my calendar.
Updating medication amounts is simply done via a modal and the values are saved to a SQLite database.
All the lists are built using HTML Content templates that I clone and add to the DOM based on the AJAX results.
I could spend hours trying to “fancy” up the front end with various animations, transitions etc but if I did that I’d probably never get a chance to actually use it.
Flask
I was already going to be learning Python for the Vector SDK so that’s why I chose Flask. The Flask server runs on the Raspberry Pi with Supervisor to make sure that Flask still runs should the Pi need a reboot or whatever.
Google Calendar
The Google Calendar API has a good sample on how to query a calendar for the current day’s events. Unfortunately Google Tasks doesn’t support the level of repetition or detail I needed which is why I use the Calendar instead. It also has a decent web/iOS/Android app so the calendar can be shared and maintained by others if needed.
Cron
Without being able to install software on Vector itself I use a crontab to schedule the script execution. The first run is set for the same time I get up every morning (6 am) and every 20 minutes until 8am. While the cron job will run and call the script the script itself has a check to that it doesn’t keep running if all the medication has been taken for the day.
SQLite
I chose SQLite because I don’t need the overhead of a database server and like Flask I’d wanted the experience of new technology. It stores:
- medication names
- current amounts
- last update date of my prescription details
- the number of times Vector had to remind me each day
Using Vector
When the cron job runs it attempts to establish a connection with Vector over WiFi. I’ve already paired the robot with the Pi.
It uses the same code the front end uses to query my Google Calendar and the SQLite database for medication data. It loops through the calendar events to see if anything is not marked as “Done” and calculates how much of each medicine I have left based on prescription rules.
The basic process is:
- Query calendar for events between 00:01 this morning and 23:59 tomorrow night
- Separate events into today_events, tomorrow_events, and done_events lists
- If there are today_events then build utterances for say_text() out lists of greetings, the event summary and various hard-coded sentence structures
- If there are tomorrow_events, build those utterances. These are typically a short reminder of what I should be taking tomorrow
- Query SQLite database for current medication amounts, build utterances based off this information and my current prescription
- Pick the animation to run based on current reminder count
- Run the animations and speak the utterances
- Increment the reminder count and store in the database
(There are other things to check too like battery_state() and connectivity with the robot).
The Videos
The First Reminder
For non-Irish speakers what Vector says is “Maidin mhaith” which is Irish for “Good Morning” – yeah, it looks nothing like it sounds! And for text to speech it looks nothing like that either. I had to spell it “Modjin Wah!” to get the correct sounds.
The Sad Reminder
This is one of the possible animations Vector will run if I still haven’t taken my medication after it has already reminded me of what I should be taking. I could have spent an a long time working on this animation but the sad eyes and dark blue worked very well from the start. Vector gets the point across quite easily.
The Angry Reminder
There’s 50-50 chance of getting an “angry” or a “sad” reminder if I still haven’t marked my medication as taken. For the purposes of recording it just happened to work out that “sad” ran before “angry”. It’s actually quite funny being threatened by a very tiny robot. I had no idea just how much force it hits the table with until I saw the recording!
The “Yay” Celebration
This is the celebration when everything has been marked as “done” for the day. As with the first reminder there’s are some non-English phrases in use and the pronunciations of them are not absolutely perfect (I apologise!).
There’s a 50-50 chance Vector will ask “Do you have enough medicine left?” or tell me exactly how much medication I actually have left in terms of what’s appropriate for that particular medicine – tablets in days because I take them daily and injections in months because they’re every 2 weeks.
The five phrases are picked randomly from a larger list in the script and in this instance the chosen phrases are:
- “an-maith ar fad” – Irish for “very good indeed”
- “shiok” – Singaporean / Malay for sort of like happiness/excitement
- “kowaies kateer” – Arabic (Gulf) for “very good”
- “bualadh bos” – Irish for “applause”
I elaborate about more about all the phrases used and the reasoning behind them a little further down this page.
The “No Reminder Required” Celebration
I added a special case for when I’ve actually taken everything before Vector has even run the first reminder. I couldn’t resist adding in the “Army Of Darkness” quote!
Fiero
In both celebration videos the “arms up” isn’t just motion for the sake of it. I learned about “fiero” in Jane McGonigal’s “Reality Is Broken”.
“Fiero is what we feel after we triumph over adversity. You know it when you feel it – and when you see it. That’s because we almost all express fiero in exactly the same way: we throw our arms over our head and yell.“
I couldn’t think of a better way to sum up living with a chronic illness.
Utterances
For the moment the utterances are built from parts of sentences that are randomly chosen to build up the full utterance. I have hard-coded lists for what I’ve classified as “greetings”, “sad words”, “angry words”, etc.
There’s also code to convert Event dates and times into more “human” sounding text where appropriate. Numbers are converted to ordinals or counts e.g. “This is your first reminder..” or “I’ve already had to reminder you twice…”
There’s a mix of English, Irish, Arabic and Singaporean in there. Some utterances are hilariously horrendously spelled in order to work with the text-to -speech system. I don’t even think they count as phonetic spelling but they sound almost right which is the most important point!
The reason for the different languages is an attempt to help deal with the depression. I’m Irish, I’ve lived in Saudi Arabia and I’ve lived and worked in Singapore. When I have a really bad day hearing the different languages is a reminder of experiences I’ve had outside of my current head space.
All of the Irish phrases relate to in-jokes with my friends and typically used in very unsupportive contexts when one of us has done something stupid so hearing them – from a robot of all things – reminds me of my friends and always puts a smile on my face.
It’s not a magic bullet, and at times there’s still a lot of work to do just to swim to the surface but hearing a word or a phrase is a reminder there’s so much more to me than what I’m fighting with physically or mentally.
Why not just use my smartphone?
Hopefully the videos show you why. Crohn’s can really be a grind and when you’re doing everything in your power to just get out of bed or simply hoping to get through to the end of every day you quickly get sick of hearing “ding” from your phone, seeing another bloody notification or getting yet more email. I’ve been using Vector for a few months now and what I’ve noticed most is a cognitive “offload” or less to worry about I suppose.
I will admit there have been days where I’ve deliberately not marked my medication as done just to see if Vector will get “angry” with me.
But there have been more days where I’ve taken them before any reminders just see the celebration dance.
If you started out reading this post having never heard of Crohn’s Disease before and come away with even just knowing the name that’s wonderful. Thank you for reading this far!
If you have Crohn’s, I know how tough it can be, hang in there. It may not always feel like it when things are tough but there’s infinitely more to you than this disease and the people who genuinely care about you know that too.
Future Work
Facial Recognition – A key part of an social interaction is eye contact and I’ve repeatedly tried to get Vector “find” my face before it starts to speak. I’ve tried a few “tricks” like have it say some kind of greeting to get people to look in its direction first or to slowly rotate on the spot with the camera raised until it finds a face.
At the moment the facial recognition is not totally consistent and quite dependent on the strength of the light sources in the room.
“snitch” – Following on from the facial recognition, I’d like to implement functionality that if Vector sees someone who isn’t me (and hopefully not a burglar) he’ll tell the person I haven’t taken my medication yet. Would I develop trust issues with a robot? LOL.
Improved Utterances – At the moment there is a lot of hard-coding with pseudo-random number decision making. It all works very well actually but it’s a bit messy to update and maintain so I’d like to try to improve the structure of it (now that I actually know a lot more Python than when I first started! :-P).
Medication Recognition – This isn’t really necessary but because of my interest in social robots for healthcare I’d like to add an additional step where I would have to show my medicine to it and have Vector verify it’s the correct dose.
Evolving Utterances – It might be an interesting project to see if utterances could be “evolved” with a genetic algorithm to get the correct spelling that produces the desired utterance. The test would ultimately be running it through a TTS and seeing if a STT (like Google Translate) recognised it as the word you wanted.