The Be My Eyes app is set to gain a new feature called the Virtual Volunteer, powered by OpenAI's recently announced GPT-4 model. This new tool has the potential to be a game changer for people with visual impairments, providing a virtual sighted assistant that can generate context and understanding for images in the same way as a human volunteer can.
Be My Eyes has been providing technology for the blind and low-vision community since 2012, connecting users with volunteers for assistance with everyday tasks like navigating airports and identifying products. The new feature, which uses GPT-4’s visual input capability, will allow the app to offer an even greater degree of independence to users and expand the use cases for the app.
Michael Buckley, CEO of Be My Eyes, stated that “in the short time we’ve had access, we have seen unparalleled performance to any image-to-text object recognition tool out there,” and added that “the implications for global accessibility are profound. In the not so distant future, the blind and low vision community will utilize these tools not only for a host of visual interpretation needs, but also to have a greater degree of independence in their lives.”
The difference between GPT-4 and other language and machine learning models is both the ability to have a conversation and the greater degree of analytical prowess offered by the technology. Basic image recognition applications can only identify what’s in front of them, while GPT-4 can extrapolate, analyze, and understand context, allowing it to offer much more comprehensive assistance.
One of the most exciting aspects of the Virtual Volunteer feature is its ability to offer instantaneous visual assistance for a wide variety of tasks. In a give example, users can send an image of the contents of their fridge to the tool, which will then not only identify what’s in the fridge, but also suggest recipes that can be prepared with those ingredients.
The new feature has undergone beta testing with a select group of Be My Eyes employees, yielding overwhelmingly positive feedback. Testers have praised the feature's functionality, particularly in one instance where a user was able to expertly navigate the railway system. The feature provided detailed information about the user's location on a map and offered step-by-step instructions on how to safely reach their desired destination.
In the video accompanying the announcement, a Be My Eyes user shows the app helping her with a number of tasks, including describing the look of a dress, identifying a plant, reading a map, translating a label, directing her to a certain treadmill at the gym and telling her which buttons to push at a vending machine.
Jesper Hvirring Henriksen, CTO of Be My Eyes, explains that traversing the complicated physical world is only half the story. There are challenges faced by visually impaired people when using screen readers to understand web pages, especially when it comes to images. “GPT-4 is able to summarize the search results the way the sighted naturally scan them—not reading every minuscule detail but bouncing between important data points.”This technology could simplify tasks such as reading news online and navigating cluttered e-commerce sites for people with visual impairments.
The Virtual Volunteer will be available to users in the coming months, and it has been hailed as “game changing” by Buckley. “Ultimately, whatever the user wants or needs, they can re-prompt the tool to get more information that is usable, beneficial and helpful, nearly instantly.”
If and when the Virtual Volunteer is unable to answer a question, it will automatically offer users the option to be connected via the app to a sighted volunteer for assistance.
The tool will be free for all blind and low-vision community members using the Be My Eyes app. You can currently register in the app to be placed on a waiting list for access.
We are excited to discover how the Virtual Assistant will compare to other similar apps in the market, such as Seeing AI and Envision AI. If it lives up to the hype, this should indeed be a game changing development.
Please share your own thoughts and hopes about this exciting development in the comments section below.