Smart headset gives visually impaired a new way to ‘see’

Researchers at the National University of Singapore (NUS) have developed a wearable assistive device that leverages Meta’s Llama models to help people with visual impairments “see” the world around them.

Called AiSee, the headphone-like gadget is equipped with a camera that operates as an artificial intelligence (AI) companion that helps users process visual information, integrating into their daily lives and even helping them return to the workforce.

Initially conceived as a finger-worn ring in 2018, AiSee’s design has since evolved into an open-ear headphone form factor. Suranga Nanayakkara, professor at the NUS department of information systems and analytics who led the research team, said this design was chosen over options such as glasses to avoid potential social stigma and, more importantly, to keep the user’s ears uncovered, preserving their natural spatial awareness through sound.

Based on user feedback, Nanayakkara said the design has been further improved, addressing issues such as hair obstructing the camera and the need for longer battery life. The latest iteration also functions as a standard headphone for music and calls, with AI capabilities available on-demand.

Nanayakkara said this dual-purpose design enhances AiSee’s utility and appeal to users. “It doesn’t make sense to have something that’s used once a day or maybe few times a week,” Nanayakkara said. “We’ve built it more as a smart headphone.”

A major turning point for AiSee was the integration of large language models (LLMs), which transformed the device from a simple object identifier into a conversational assistant. This allows users to not only identify an object but also ask follow-up questions about it.

The device runs an agentic AI framework where computer vision and reasoning models work in tandem with a Llama model to understand a user’s intent and execute tasks. To run Llama efficiently on the Android-based device, the team used quantisation techniques to reduce the model’s size to between one and three billion parameters.

Hussel Suriyaarachchi, AiSee’s chief technology officer, said the agentic framework is also flexible enough to incorporate newer Llama versions as they become available. “If it works with Llama 3.2, we can easily replace the model with newer versions because the underlying architecture of Llama is similar,” he added.

Nanayakkara, who is also co-founder of AiSee, the eponymous startup behind the device which was spun off from NUS about a year ago, said the decision to use Llama was driven by feedback from the company’s visually impaired employees, particularly the need for offline processing of sensitive documents.

“If you are a blind person getting a new employment contract, you’d want to understand what’s in the contract and ask questions about it. You don’t want it to be scanned and uploaded to the cloud,” Nanayakkara said. “That’s why we chose to use Llama, especially the smaller models, that can run on our device.”

AiSee is being supported by the NUS graduate research innovation programme and has received funding from impact investors. The company, which won the Meta Llama Impact Innovation Award in 2024, is in talks with organisations such as museums and airport operators to make their spaces more inclusive and plans a consumer launch soon. It is also working with Southeast Asian super app Grab to create a voice-based system for booking rides.

For now, AiSee has not developed support for local languages beyond what’s available through Llama, though it recently received a request from a foundation in the United Arab Emirates on localisation options. “That’s a consideration for us depending on the resources they have,” Nanayakkara said.

Nanayakkara sees a future for AiSee beyond assistive technology, where it can serve as a visual intelligence tool for everyone. “We foresee this not as a device for the blind,” he said, citing the curb-cut effect where features designed for people with disabilities tend to benefit the general population once constraints are removed. “The commercial success of AiSee will not be as an assistive technology, but as a hands-free and screen-free way to do effortless computing.”