Goal

Samson is a concept project I introduced to my innovation team while working at Memorial Sloan Kettering, a large oncology center in New York City. As if cancer isn't already enough of a burden, imagine a child with cancer. In pain, fear, and confusion, unable to express themselves to convey what is wrong and what hurts, and alone. Sometimes children might need to be isolated after a bone marrow transplant.

Meet Samson

So, meet Samson the cat, a virtual companion that can accompany a child during their journey at the hospital and even after the leave. But unlike an app or a toy, my team and I thought of Samson as an embedded AI, a ghost in the shell brought to life by any available technology. This makes Samson largely device and technology-independent. A phone, a screen, a toy, a creature living in a metaverse, while also capable of taking on a physical form through any IoT technology available. This creates a sense of presence and continuity. I conceived and designed certain aspects of Samson to offer physical, behavioral, and emotional support, a jack of many trades - providing distraction, guidance, and motivation, providing a voice, and being a guardian to the child. Although a concept, what follows are some actual solutions, I built with the help of my team.

3 - Traditional Screens

Again, by being technology-independent, interactions between the child and the virtual companion gives the companion a presence, continuity, but also agency. This facilitates bonding and inspires play and meaningful interactions between Samson and our younger patients. But it also helped me rethink how to interact with a technology like it. Starting from the more traditional, Samson be rendered to a screen, and voice and gesture commands can be used to interact.

4 - Wall based projections

Thinking ever bigger, we can imagine Samson occupying the 2D world of hospital corridors, walls, doors, ceilings, and floors. Well-designed systems of projectors and cameras can capture voice and full-body gestures and actions.

Metaverse

Further still, we can invite the child to enter a virtual world. Within this world, voice, gestures, and VR-specific handheld controllers can achieve interaction with Samson. To give you an idea of such a VR experience, here is a short demo of myself in Samson's virtual home world. The Oculus Quest no longer strictly requires users to wear the controllers. Instead, cameras can capture my hands in real-time and translate their movement into gestures on screen. Samson's AI understands a core number of interactions and can play several gesture-based games, such as peek-a-boo or Simon says. Here, I exchange some simple gestures and community my feelings, which evokes an appropriate response from Samson.

Physical Rehabilitation

When combined with motion capture, one practical application of this gesture-based interaction is an interactive physical rehabilitation scenario. Samson can provide positive feedback by joining her in exercise, making the physical rehabilitation exercises less daunting, tedious, and fun.

Device

Finally, Samson can also be a physical device taken home by the child to monitor their behavioral health as they recover and move forward. I built a prototype, combining IoT hardware and LEGO. Such a device can be situated to observe the child play, study, interact and sleep. Although perhaps somewhat invasive, my conversations with the neuropsychological staff at Memorial Sloan Kettering did reveal a need for such a tool. Children having gone through the process of cancer treatment often show a host of neuropsychological setbacks compared to their healthy peers. Depending on the child's age, these can amplify and create significant behavioral and learning problems down the road. A monitoring system was seen as most welcome, especially if it produced metrics around the child's focus, mood, sleep patterns, and interactions with siblings and parents.

Technologies Used