That composition sounds lovely (Image: Pranav Lal)
Imagine taking photographs you can hear – thanks to sonification technology that translates the visual world into sound.
Pranav Lal, a cybersecurity expert from Delhi, India, was born blind and he’s been using technology called the vOICe to take holiday snaps, most recently while trekking in the Himalayas.
The technology was designed to. But Lal where he picks out the most pleasant sounding composition. “Scenes with varied elements sound interesting, for example, a bridge with lots of cables or a bright natural setting with streams and rocks,” he says. “It’s like an image orchestra.”
The vOICe transmits an image from a camera to computer software to be turned into sound. In Lal’s case, the camera part was a smartphone upgraded with a fisheye lens strapped to a sports headband.
Position and pitch
After converting the image to greyscale, the software uses different characteristics of sound to represent visual features as it pans across a scene from left to right. The position of an object is denoted by pitch – a feature higher up in the image has a higher pitch – and brightness is mapped to loudness.
Lal admits that the complex shapes that inspire him can be a challenge to identify, but he can usually distinguish the foreground and background of an image by separating the shapes in his head. Having used the system for a while, he can now create a mental image based on the soundscape he hears.
But the result is different from the way he pictures shapes when feeling an object. “I’ve been told that it sounds like I am describing the negative of a photograph,” he says.
Making sense of things
The vOICe “rewires” the senses. After a period of training, the brain begins to process the soundscapes created by the system in the visual cortex, treating them similarly to images.from the University of Bath, UK, has been working with blind people who use the system to investigate how visual parts of the brain are able to substitute input from other senses.
In a recent study, he found that people using the vOICe can recognise blurry shapes just as well as people with typical vision, even though computer vision systems seem to struggle to identify such objects.
Video: Blind man takes sonic holiday snaps
The basics of the vOICe are surprisingly easy to pick up, as Proulx demonstrated last month atin Southwold, Suffolk, in the UK (see video, above). After hearing the sonification of a few basic lines and shapes, a volunteer who had his eyes covered was able to read words just by listening to the corresponding sounds.
Later this year, a game version of the vOICe will be launched as a smartphone app to help train people to use the system as well as to promote it. Small improvements are constantly being made, most recently the ability to email photos. Over the years, the technology has become wearable and can be used with bone-conduction headphones, which don’t cover your ears so you can still hear what is going on around you.
As well as taking photos, Lal has used the system tosent back from space last month. “I could make out mountains, but they were hazy,” he says. “A patch on the right of the image was clearer as it was better lit.” The vOICe, which has a setting for colour detection, spoke the colours in the image.
Although it is not really essential for navigation – the original purpose of the vOICe – the ability to identify colours can be put to use. By selecting which colours to sonify, a user can focus on aspects of a scene, for instance, if you wantapples at the shop, you can change the setting to only hear the red ones, explains Proulx.
Different cameras can also be hooked up to the device to augment human vision. For example, using a thermal camera can replicate the vision of a snake through sound, or a night vision camera can be used to “see” in the dark.
Lal would like to delve into night photography during his travels, but he has not been able to find an affordable and portable night vision camera. In October, he will be travelling to Thailand where he plans to wear the vOICe constantly to take photos.
This entry passed through the Full-Text RSS service – if this is your content and you’re reading it on someone else’s site, please read the FAQ at fivefilters.org/content-only/faq.php#publishers.