The Kinect is dead; long live the Kinect

7 months ago by Lucas Croft

From experimental games to medicine and ghost-hunting, the Kinect's seen plenty of weird application in the last seven years.

Hark back, for a moment, to the late 2000s. The Wii’s strong performance had the gaming industry’s eyes fixed firmly on a future defined by motion controls. In 2010, Sony released its Wiimote equivalent, the PlayStation Move, whose functionality mirrored the Wii in many ways. Microsoft, on the other hand, took a much different approach, completely eliminating the need for a controller. “You are the controller,” proclaimed the 2009 announcement trailer for what was then called Project Natal.

Much like the Wii, Kinect’s novel approach caught on quickly with consumers, selling eight million units in its first 60 days on the market. Overtime, however, the peripheral lost momentum due to unmet promises and a dearth of interesting games. The motion detection never quite worked as well as advertised, voice controls were frustratingly inconsistent, and features that were announced in 2009 never made it into the home. The Kinect finally met its demise after Microsoft unbundled the accessory from its Xbox One units due to struggling Xbox One sales.

While most will remember the Kinect for its meteoric rise and equally catastrophic downfall, developers found interesting ways to leverage the technology. Limited by somewhat unreliable technology, the best ideas for Kinect Games relied not on perfect one-to-one input, but rather novel ideas that maximize the fun factor.

The Gunstringer (Twisted Pixel, 2011)

The Kinect elevated Twisted Pixel’s debut retail title, The Gunstringer, turning a simple concept into an action-packed, 3D run ‘n gun. Set in the Wild West with an art style similar to Grim Fandango, players controlled a finger pistol wielding marionette sheriff seeking revenge against his murderous posse. The title settled into the Kinect’s sweet spot of taking a concept that wouldn’t generally move the needle and adding just enough novelty to set it apart from other offerings. Every fan of Westerns spent at least some portion of their childhood wielding imaginary six shooters, and The Gunstringer played perfectly on that nostalgia.

Project Natal’s reveal trailer also showcased an object scanning feature that allowed players to scan their real-world objects into the game, a concept Kinect Fun Labs toyed around with in its Build-A-Buddy minigame. Players could scan in any object, design a custom personality for the object, and then the game would animate the object with the player-built personality. At most, the minigame served as a proof of concept for Kinect’s ability to render an object simply by scanning two images of it, even though the concept never made its way into other games as advertised.

FRU was the perfect embodiment of the Kinect's best days, and what the peripheral could've been.

Kinect’s best game for either iteration waited until its dying moments to launch. FRU, the 2D platformer from Through Games, turned players into a red lens decoder, revealing hidden objects to help their avatar navigate a level. Motherboard’s Emanuel Maiberg described FRU as “a kind of platformer-yoga hybrid,” summing up what made the game so appealing for the few remaining Kinect users. It’s simple, yet novel premise appealed to both long-time fans of the platforming genre and people who have never picked up a controller in their lives. At the end of the day — and the platforms life-cycle — FRU was the perfect embodiment of the Kinect’s best days, and what the peripheral could’ve been.

FRU (Through Games, 2016)

The game only sold 10,000 units, but developer Mattia Traverso seemed to take it in stride. "I saw Kinect as a way to experiment and make cool shit," he said to Engadget. "We never really thought that we were gonna use Kinect to make money."

Upon the Kinect’s release, the idea of people hacking the Kinect to serve purposes outside of what Microsoft intend made the company uneasy. However, while Microsoft obviously wanted the Kinect to be central to the Xbox experience, the peripheral gained a bit of a second life beyond video games.

The company wisely saw the potential of Kinect in third-party development and took a more active approach in incubating the creative uses of their product. It branded the initiative “The Kinect Effect,” allowing companies, research groups, and other third-party developers a more sanctioned look under the hood of the motion sensor. The push brought a bevy of highly creative projects to life, to varying degrees of success.

When a crew from MIT dug into the Kinect, it developed a UI interface inspired by the movie Minority Report. With the ability to detect all five fingers and the palm, the graphical interface and hand detection software was able to detect motion “in a cloud of more than 60,000 points at 30 frames per second, allowing natural, real time interaction,” according to the creators from the Robot Locomotion Group and Learning Intelligent Systems teams.

The technology never made it out of Cambridge, MA. Perhaps the broad application of such a system was too far ahead of its time or, more likely, too impractical for wide adoption. However, it showed just what made the Kinect so intriguing upon its initial reveal. The ability to manipulate technology through simple gestures made the device feel like something otherworldly, a foray into technology from the distant future. But like all revolutionary ideas, it took a perfect combination of concept and context to really find its stride.

At the 2011 Pacific Health Summit, Microsoft’s Chief Research and Strategy Officer, Craig Mundie spoke to Kinect’s potential uses in hospitals, doctors offices, and healthcare clinics across the country. During his presentation, Mundie painted a picture of a world where the Kinect revolutionized both mental and physical healthcare. This vision of the future proved more than just a pipe dream from one of Microsoft’s c-suite executives: the Kinect caught the attention of hospital CIOs around the globe.

The ability to manipulate technology through simple gestures made the device feel like something otherworldly, a foray into technology from the distant future.

The Kinect’s potential in telemedicine also brought with it promises of lower healthcare costs. A study published in 2013 by the International Journal of Electronic Finance suggested that the device could cut the US healthcare bill by $30 billion. The logic followed that using the Xbox Kinect along with other affordable technology — a laptop, an Azure connection, and Office 365 — would allow doctors to meet with patients remotely, replacing costly teleconferencing technology that healthcare centers often relied on.

The study envisioned a world where the Kinect would help hospitals cut down on the $10 billion spent annually treating hospital-acquired infections. According to the team, "The Kinect allows doctors to control the system without breaking the sterile field via hand gestures and voice commands with a goal of reducing the direct cost of healthcare associated infections to hospitals and patients.”

A composed photo from the GestSure website.

It is here that the team at GestSure recognized the Kinect’s potential as a gesture-driven controller for certain UIs, helping MIT’s previous vision find its appropriate context. Through the simple use of hand gestures and voice commands input to a computer through an Xbox 360 Kinect, surgeons could use GestSure to consult medical images in the middle of surgery. The technology allowed surgeons to skip time-consuming steps to scrub out, check an image, and scrub back into surgery, a process that can cost as much as $620 each time it has to be done.

Health-care provider Kaiser Permanente partnered with software developer Vectorform to develop an autism assessment tool called Monkey Business. Previously, in order to identify autism, a therapist would have to sit with the child and ask a series of questions. Young children, even those that are not on the autism spectrum, have a tendency to become bored through the process, making it difficult to complete the assessment. Monkey Business, took the laborious task of administering the assessment and used a fun monkey avatar to engage the children, instead.

The University of Minnesota’s Institute of Child Development took a different, more passive approach to identifying autism in children. A series of Kinect sensors connected to computer vision algorithms would monitor a child’s activity level and compare them to an average. The system identified children with much lower or much higher activity levels as possibly autistic. Researchers hoped that it would allow a child to receive intervention therapy for speech and communication skills earlier.

From the physical to the metaphysical, the Kinect became a prominent tool among ghost hunters, catching the attention of the League of Occult Research and Education (LORE) following the release of the Xbox One. In a blog post titled “New Xbox One Kinect 2 is Advanced Ghost Hunting Tech,” LORE discusses how Kinect 2.0’s IR sensor, heartbeat monitor, and 3D depth mapping make it the perfect camera for activity from the spirit world, claiming the device “might just be the most advanced ghost friendly device ever devised.”

From Travel Channel's Ghost Adventures From Travel Channel's Ghost Adventures (2012).

YouTube videos claiming to show “Kinect Ghosts” flooded the platform following the release of Kinect 2.0, as the camera registered users that weren’t there. While Microsoft never sanctioned this functionality, even official Microsoft channels have shared stories of their use for this purpose. In 2016, the Kinect for Windows Product Blog reported a case where a team of ghost hunters used the Kinect sensor to explore a haunted house near the Miami International Airport.

Call it glitched technology with an added dose of confirmation bias, if you will, but ghost hunters around the world have widely accepted that the Kinect is sensitive to ghosts and other spiritual entities. The Ghost Hunter Store believes in the application so much that it still sells a modified Kinect linked to an eight inch tablet for $399, claiming it “also seems to see bodies when there is nothing there the naked eye can see,” ultimately asking the poignant question. “Spirits?” The store also offers a mobile version of this technology for $379.

In perhaps the most unlikely spot for product placement, the “Kinect Ghost” phenomena even found its way into pop culture in the horror movie Paranormal Activity 4. In the trailer for the film, a character demonstrates “something really cool with the Kinect that you can do,” and proceeds to invert the devices infrared grid of dots so that it is visible. The Kinect is then able to pick up the movement of the apparition that haunts the family.

While the Kinect’s use in Paranormal Activity 4 does not represent an actual feature of the device, both its inclusion in the film and ghost hunters’ adoption of the technology  show just how far and wild the Kinect frenzy grew shortly after launch. In both the practical and far-reaching senses, the Kinect started a strong, albeit short-lived, craze.

There’s no telling how high the Kinect’s ceiling could’ve gone given the perfect circumstances. However, after seven years, its underperformance as a commercial product led Microsoft to abandon the hardware; the Kinect never left the launch pad, a victim of both its own underdeveloped features and Microsoft’s bullish strategy packaging it with the original Xbox One. These unique projects show both the potential of the peripheral and the range of applications it could have had when released into the wild. When measured as a whole, the Kinect was an unequivocal failure, but in certain sectors and in the hands of creative developers, the technology was able to accomplish even more than Microsoft ever envisioned.