Novel smartphones apps using Artificial Intelligence (A.I.) are really useful in making visual information accessible to people who are blind or low vision. For instance, Seeing A.I. or TapTapSee allow you to take a picture of your surroundings, and then they tell you what things are recognised, for example, “a person sitting on a sofa”. While A.I. recognises objects in a scene if they are common, at the moment these apps can’t tell you which of the things it recognises is yours, and they don’t know about things that are particularly important to users who are blind or low vision.
Using A.I. techniques in computer vision to recognise objects has made great strides, it does not work so well for personalised object recognition. Previous research has started to make some advances to solving the problem by looking at how people who are blind or low vision take pictures, what algorithms could be used to personalise object recognition, and the kinds of data that are best suited for enabling personalised object recognition. However, research is currently held back by the lack of available data, particularly from people who are blind or low vision, to use for training and then evaluating A.I. algorithms for personalised object recognition.
This project, funded by Microsoft A.I. for Accessibility, aims to construct a large dataset by involving blind people.
– The ORBIT (Object Recognition for Blind Image Training) Dataset project page
How do you construct a “large dataset”? And do so with the accessibility credo “with us, not for us”? Well, you need a camera app that people will want to use, that then talks to a dataset collating back-end. In particular, the user-experience challenge of a camera for the blind, and research-ethics grade data infrastructure. That’s what I was brought on to make.