The Internet of Things has a memory problem. No, it’s not the limited amount of digital storage in the connected objects quickly filling homes, industrial plants and cities — it’s the limit of the human brain to process abstract symbols.
It’s not a new problem, either. Since the days of the text-only “command line”, interface designers have been challenged by the imbalance between the nearly infinite number of commands a computer can interpret, and the very limited selection that a user can hold in her brain at one time. The graphical user interface, or GUI, gave us an intuitive metaphor — the desktop — for representing and navigating through the possibilities, and we’ve transferred that convenience to the mobile world with apps full of icons and menus.
But Valentin Heun, a doctoral student at the MIT Media Lab who studies interface design, says that the GUI hits its limits when we try to extend its reach beyond the borders of the screen.
A smart light bulb might have millions of color options, thousands of brightness settings and hundreds of blinking or color-changing patterns to choose from; and it might be only one of dozens of connected objects in your home. In choosing the right icon from the right menu from the right app, a user is burdened with interpreting layer after layer of “these abstract representations that try to give you a hint of what it is linked to in the physical world,” Heun says. “Then you stand in the kitchen and you just want to switch on a light.”
Flip. On. Flip. Off. The lightswitch is a paragon of elegant simplicity, a basic binary counterpoint to the combinatorial complexity of the smart bulb. And that tension, between what is possible through programming and what is easy to do with your hands, is at the heart of Heun’s work. His goal is to “push back the GUI as much as possible” by separating an object’s digital programmability from its physical operation.
“There are a ton of interactions that live perfectly in the physical world,” Heun says, like on-off switches and volume knobs — the stuff you need immediate, unthinking access to every time you use a device. The more complex behaviors — like bulb colors, playlists, and automation rules — are things you set once, or only modify occasionally. For those, Heun and his Media Lab collaborators built Reality Editor (now part of the Open Hybrid platform).
Instead of using your phone or tablet as a container for apps and icons, Reality Editor makes it a transparent window into the world of digital customization. Hold your phone up so the camera points at a smart device and the app will display a virtual control panel that hovers next to the object, offering whatever settings and options that device allows. You’ll also see nodes corresponding to the physical controls the device offers, and you can create interactions between devices by linking these together with a swipe of a finger.
Want the switch on your desk lamp to control the overhead lights? Swipe. Want to use the volume knob on your Bluetooth speakers as a lighting dimmer instead? Swipe swipe. The beauty of Reality Editor is that making a change to one of your smart objects no longer requires finding the right app, menu, and toggle — it just requires finding the object itself.
Rather than treating our phones as bloated remote controls, Reality Editor lets us leave them in our pockets until we need to fix or tinker with something. “You can think of your phone as a multitool for physical objects,” Heun says.
“I want to see that in 50 years, you will be saturated with objects that can have such functionality,” Heun says. “We have a platform that breaks it down so simply that in 50 years you’ll have an object, and we can guarantee that it will still be able to interact with objects you have now.”
Like many tech projects spawned from within academia, the entire Open Hybrid platform is free to use and open-source, so the rest of us can build upon it as we see fit.
“This is an amazing, empowering platform because it allows a web designer to suddenly jump into augmented reality without learning anything new,” Heun says, and the same is true of Arduino tinkerers. Just having the opportunity to create something that supports the explosion of creativity in DIY and IoT technologies, and helps to spread them into the physical world, “is already a big win,” he says. “You can create something that is meaningful not only for yourself, but for the whole technological community.”