Apple Previews New Accessibility Features Coming This Year, Including Expanded Image Recognition Capabilities for VoiceOver Users

Member of the AppleVis Editorial Team

Ahead of Global Accessibility Awareness Day on May 20, Apple has previewed a number of accessibility Features coming later this year for those with mobility, vision, hearing, or cognitive disabilities.

“At Apple, we’ve long felt that the world’s best technology should respond to everyone’s needs, and our teams work relentlessly to build accessibility into everything we make,” said Sarah Herrlinger, Apple’s senior director of Global Accessibility Policy and Initiatives. “With these new features, we’re pushing the boundaries of innovation with next-generation technologies that bring the fun and function of Apple technology to even more people — and we can’t wait to share them with our users.”

The new features include expanded image recognition capabilities for VoiceOver users, AssistiveTouch for Apple Watch, new eye-tracking hardware support for iPad, new background sounds to help minimize distractions, and for those who are deaf or hard of hearing, Made for iPhone (MFi) will gain support for new bi-directional hearing aids.

Explore Images with VoiceOver

Here's how Apple describes the improved image recognition capabilities that will be coming later this year:

Building on recent updates that brought Image Descriptions to VoiceOver, users can now explore even more details about the people, text, table data, and other objects within images. Users can navigate a photo of a receipt like a table: by row and column, complete with table headers. VoiceOver can also describe a person’s position along with other objects within images — so people can relive memories in detail, and with Markup, users can add their own image descriptions to personalize family photos.

AssistiveTouch for Apple Watch

To support users with limited mobility, Apple is introducing a new accessibility feature for Apple Watch:

AssistiveTouch for watchOS allows users with upper body limb differences to enjoy the benefits of Apple Watch without ever having to touch the display or controls. Using built-in motion sensors like the gyroscope and accelerometer, along with the optical heart rate sensor and on-device machine learning, Apple Watch can detect subtle differences in muscle movement and tendon activity, which lets users navigate a cursor on the display through a series of hand gestures, like a pinch or a clench. AssistiveTouch on Apple Watch enables customers who have limb differences to more easily answer incoming calls, control an onscreen motion pointer, and access Notification Center, Control Center, and more.

Eye-Tracking Support for iPad

Later this year, iPadOS will be gaining support for third-party eye-tracking devices that will enable people to use their eyes to navigate and control an iPad. compatible MFi devices will track where a person is looking onscreen and the pointer will move to follow the person’s gaze, while extended eye contact performs an action, like a tap.

Background Sounds

Apple says that “everyday sounds can be distracting, discomforting, or overwhelming,” so in support of neurodiversity it is introducing new background sounds.

According to Apple this is how background sounds will help minimize distractions and help users focus, stay calm, or rest:

Balanced, bright, or dark noise, as well as ocean, rain, or stream sounds continuously play in the background to mask unwanted environmental or external noise, and the sounds mix into or duck under other audio and system sounds.

Made for iPhone Hearing Aids and Audiogram Support

In an update to its MFi hearing devices program, Apple is adding support for new bi-directional hearing aids. The microphones in these new hearing aids enable those who are deaf or hard of hearing to have hands-free phone and FaceTime conversations.

Apple says that the next-generation models from its MFi partners will be available later this year.

Apple is also bringing support for recognizing audiograms to Headphone Accommodations. This will allow you to customize audio with your latest hearing test results imported from a paper or PDF audiogram.

Addition New Facilities and Features

Ahead of the software-dependent features, Apple is launching on May 20 a new facility called SignTime that aims to make it easier for those with speech disabilities to communicate with Retail Customer Care and AppleCare specialists:

This enables customers to communicate with AppleCare and Retail Customer Care by using American Sign Language (ASL) in the US, British Sign Language (BSL) in the UK, or French Sign Language (LSF) in France, right in their web browsers. Customers visiting Apple Store locations can also use SignTime to remotely access a sign language interpreter without booking ahead of time. SignTime will initially launch in the US, UK, and France, with plans to expand to additional countries in the future.

And More

Apple says that additional features coming later this year include:

  • Sound Actions for Switch Control replaces physical buttons and switches with mouth sounds — such as a click, pop, or “ee” sound — for users who are non-speaking and have limited mobility.
  • Display and Text Size settings can be customized in each app for users with colorblindness or other vision challenges to make the screen easier to see. Users will be able to customize these settings on an app-by-app basis for all supported apps.
  • New Memoji customizations better represent users with oxygen tubes, cochlear implants, and a soft helmet for headwear.

And to Mark Global Accessibility Awareness Day

To celebrate Global Accessibility Awareness Day, Apple is launching new features, including an Accessibility Assistant in Siri Shortcuts, Today at Apple sessions in American Sign Language, developer spotlights in the App Store, a new collection in the Apple TV app, reading recommendations in Apple Books, and new guides in Apple Maps from Gallaudet University.

  • This week in Apple Fitness+, trainer and award-winning adaptive athlete Amir Ekbatani talks about Apple’s commitment to making Fitness+ as accessible and inclusive as possible. Fitness+ features workouts inviting to all, from trainers using sign language in each workout to say “Welcome” or “Great job!,” to “Time to Walk” episodes changing to “Time to Walk or Push” for wheelchair workouts on Apple Watch, and all videos include closed captioning. Fitness+ also includes a trainer demonstrating modifications in each workout, so users at all levels can join in.
  • The Shortcuts for Accessibility Gallery provides useful Siri Shortcuts for tracking medications and supporting daily routines, and a new Accessibility Assistant Shortcut helps people discover Apple’s built-in features and resources for personalizing them.
  • Today at Apple is offering live, virtual sessions in ASL and BSL throughout the day on May 20 that teach the basics of iPhone and iPad for people with disabilities. In some regions, Today at Apple will offer increased availability of Accessibility sessions in stores, through May 30.
  • In the App Store, customers can read stories about Lucy Edwards, an influencer on TikTok who is blind and shares her favorite accessible apps; App of the Day FiLMiC Pro, which is among the most accessible video apps for blind and low vision filmmakers; and more in the new Express Yourself Your Way collection.
  • The Apple TV app will spotlight its Barrier-Breaking Characters collection which celebrates authentic disability representation onscreen and behind the camera. It features guest curation from creators and artists like the cast of “Best Summer Ever,” who share their favorite movies and shows in an editorial experience designed by American Pop-Op and Urban Folk artist Tennessee Loveless, known for his vibrant illustrations and colorful storytelling told through the lens of his colorblindness.
  • Apple Books adds reading recommendations from author and disability rights activist Judith Heumann, along with other themed collections.
  • Apple Maps features new guides from Gallaudet University, the world’s premier university for Deaf, hard of hearing, and Deafblind students, that help connect users to businesses and organizations that value, embrace, and prioritize the Deaf community and signed languages.

Apple hasn't given specific dates for the accessibility features it has previewed today, but it seems safe to assume that they will come with this fall's major operating system updates.

We would love to hear your opinions and expectations for these upcoming features, so please share them by posting a comment below.

Options

Comments

Submitted by Holger Fiallo on Wednesday, May 19, 2021

Nice but is better that apple focus on addressing bugs and updating voiceover instead of ading nice features. I do not believe voiceover has a major upgrate for a long time. Many people have to deal and cope with bugs that make the experience of using apple products frustrating.

Submitted by Amir on Wednesday, May 19, 2021

Member of the AppleVis Blog Team

In reply to by Holger Fiallo

@Holger Fiallo IMO VoiceOver received a major update with iOS 14, and since it's quite feature-rich, there's really no need to provide major updates for it. Of course, fixing bugs is horse of a different color but to say that VoiceOver needs major updates distorts the actual picture. It offers lots of features, is quite customizable, and works well with physical keyboards. It's IMO as mature, or even more mature, than screen readers for Windows. Yes -- I want stuff like third-party TTS support for VoiceOver and more non-English/Persian voice options in iOS 15, but this makes me move into the rambling territory. 😀

Features are cool but what new major upgrate happen> Yes we got control voice and other things but voiceover itself? Tapping the back phone to start a shortcut is nice but really how many people use it? I do not. I want voiceover to be better specially with the voices. I had use iPhone since 4. I used 4s, 5, 7, x, 11 pro max and currently 12 pro. I have apple watch series 5. My first PC was the apple C PC. Again features is nice but I want the voice to be better.

Submitted by Christopher on Wednesday, May 19, 2021

In reply to by Holger Fiallo

I couldn't have said it better myself.

Submitted by Greg Wocher on Wednesday, May 19, 2021

Hello,
While the VO image features are nice and innovative they don't address the big issues with VO on iOS or the Mac. VO needs serious bug fixes on both platforms. Braille really needs some major bug fixes. There are features, those of us who develop on the Mac, have been asking for for years and they just get ignored. No VO on the iOS is no where near as mature as Windows screen readers, neither is VO on the Mac for that matter. VO does work well on iOS but it has a lot of bugs that need fixing before they add these new features. I love Mac OS and iOS but VO really needs some fixes.

Submitted by Dennis Long on Thursday, May 20, 2021

I agree with Amir Voice over is extremely powerful and is not in need of a rewrite that is to far over the top. It is better then android will mostly likely ever be. You have excellent keyboard support and braille support is properly built in and not via a third party add on.

Submitted by Gruia on Thursday, May 20, 2021

I want support for trackpad on ipad

Submitted by Maldalain on Thursday, May 20, 2021

I do not want new future, they are just bad omen for me. I need Apple to fix the already available features. Braille has been broken for very long time, text editing is glitchy with longer texts, browsing the web could be improved, more customizability is welcome!

Submitted by Andy Lane on Thursday, May 20, 2021

To be frank the state of such a basic feature as text editing on any platform that claims to be usable in any meaningful way is comical. We pay thousands for these devices just like everyone else but editing a document is frustrating to impossible? Get real Apple. While I’m at it I want proper volume control for both the on device speaker and the Bluetooth connected speaker simultaneously without the on device speaker blowing my ear hole out every time I play content on the Bluetooth one.. These bugs should be show stoppers for any future purchases but we put up with it and smile, its not good enough. The text editing issue has existed for years now and the Bluetooth volume for 1 - 2 years. These should be fixed with the next release they are so fundamental to the user experience. To those suffering the Braille issues I’m so sorry for you. You’ve paid thousands like me for basic advertised functionality thats essential to you but can’t use it? That should be and is unacceptable and apple need to step up to the plate once and for all on these issues. No way they’d get away with it if it wasn’t accessibility related. That means we have a 2 tear system of priority. I know thats uncomfortable but its true and its not okay. anyone claim its not that way, I’d be interested to hear the justification/rationalisation. Sometimes its justified to get emotional about things that matter.

Submitted by Oliver Kennett on Thursday, May 20, 2021

I'm interested in the assistive touch. I sold my Apple Watch because I think they are over priced nonsense. If you want health tracking, get a Fitbit otherwise anything you do on the watch is a worse experience than doing it on the iPhone.

Assistive touch could be cool though, mid air finger tapping to activate processes, simplest being telling the time or, still better, controlling navigation. With a dog in one hand, or a cane, having to stop to fumble with an Apple Watch isn't ideal.

Regarding bugs... Yes, I completely agree. I think publicity is a big motivator here because you can't advertise the fact that apple have failed in certain aspects of actualising VoiceOver.

My big gripe is the VO F function and how it gets stuck and only reads the search string. It should cycle to the top. Reported it a couple of years ago, still nothing. It's the fastest way I've found to navigate, and it's broken.

Submitted by mmote on Thursday, May 20, 2021

I'm thankful that I can use my iDevice the way that I can. I applaud Apple for what they are doing with respect to Accessibility. Sure, we have a lot of issues, but I can tell you, as a PC user and and AT professional, we have lots of issues on other platforms. And they have been around considerably longer too. Accessibility and useability are constatnly moving targets, and the challenge is to keep up with the growing technology demands and the needs of all consumers who use them. I know Apple has disppaointed us all over the years, but overall, I think their accessibility teams do good work.

Submitted by Carter Wu on Thursday, May 20, 2021

In fact, the function sounds really attractive, especially tables like receipts can also be browsed by row and column, but for users who live in China like me, these functions may be extravagant, because nowadays VoiceOver’s image descriptions can only work in English, not the Chinese that we are familiar with.Some time ago I participated in an event held in Sanlitun Apple Store in Beijing. There were engineers from the image description team present. They were very confused about whether Chinese users need image descriptions,So I think the newly added functions will probably not support Chinese, and it is still difficult for us to use them.

Submitted by PaulMartz on Thursday, May 20, 2021

Member of the AppleVis Blog Team

I'm really excited about the ability to OCR a receipt and navigate it as a table. This has been a long-standing OCR issue and I'm glad to see Apple step up and fix this obvious shortcoming. Yay!

That being said, if Apple really wanted to make a statement for Global Accessibility Awareness Day, they would announce a redoubled effort to address issues in existing accessibility features. If a city built new streets but never filled potholes in the old streets, the citizenry would vote out the mayer and city council so fast their heads would spin.

Submitted by Tayo on Thursday, May 20, 2021

I'm sort of two minds about this. Personally I think the idea of improved image recognition is great; this might even improve the accessibility of some apps that are heavy on the images. However, VO on the Mac in particular is lagging behind the competition. At this point the number of bugs is probably to the point where they would haave to expand the accessibility team just to have a subset of people devoted to squashing VoiceOver bugs of long standing. Also, note that no new VoiceOver improvements seem to be coming to the Mac this year. This is unfortunate, because Macs themselves are becoming more powerful with every update, and the potential app gains from using the M series chips are huge. So while the new feature is great, VO for the Mac really needs some love and attention.

Submitted by Nikola Jovic on Thursday, May 20, 2021

Hello,
nobody said that bugs won't be fixed. However, in a big announcement like this, typically you announce the most exciting new feature, not a bugfix. I expect bugs to be resolved to a reasonable degree, but this isn't something announced in advance, for sighted or blind users. So, the fact we are getting a new feature doesn't mean bugs won't be fixed.

Submitted by Chris on Thursday, May 20, 2021

The problem seems to be that iOS is being prioritized over macOS. I suspect this is because the accessibility team isn't large enough to cover both platforms competently and/or someone has decided that iOS is the more popular system, thus more resources will be dedicated to it. This is beyond unacceptable for a company with the resources like Apple. They could afford to employ many more engineers to fix the Mac and make VoiceOver live up to the potential I know is there. Sadly, I don't expect this to happen anytime soon. I tried writing a message to Tim Cook concerning the horrible state of accessibility on the Mac and was treated to a canned response from Apple Accessibility.

The problem with Apple is their "We're going to do whatever we want" attitude. No one from Apple accessibility really engages with us, the very community they claim to care so much about. If they care so much, we should be able to have open dialogue and ask the hard questions such as "Why is VoiceOver on the Mac in such a horrendous state that it makes an open-source project like NVDA make it look like a joke? Why does your public PR make everything seem absolutely positively fantastic when the opposite is true? Why hasn't Tim Cook made a public apology for this situation and pledged to rectify the situation immediately? Why are we deemed unworthy to speak directly to Apple employees?" Apple's culture really pisses me off! I don't care about image descriptions! I want bug fixes, enhancements to text editing, accessible remote support on the Mac, etc. In short, I expect a lot more from the pioneer of mainstream accessibility. I'm tired of sugarcoating things. As has also been stated, we pay the same amount of money for these products so are entitled to receive as close to a comparable experience as realistically possible. In the case of the Mac, we definitely aren't, which is why I cannot justify purchasing another one.

I'm also sick and tired of the Apple apologists that constantly say everything is fine, Apple cares about the Mac, bugs are fixed, you're being ungrateful, Apple engages with us, blah blah blah. This doesn't help the situation. Those who have great contact must be privy to special insider access for whatever reason which again doesn't really help. I truely believe Apple can make great things happen. They've demonstrated it in the past. It angers and saddens me to observe this blatant neglect.

Submitted by Holger Fiallo on Thursday, May 20, 2021

In reply to by Nikola Jovic

How soon people forget iOS 13 and the many bugs and issues with voiceover. The issue is not new features but stability and easy to use iOS to enjoy the experience of the apple products. Voiceover it self continues to be an issue for many and saying the bugs will be fix is not an answer. I pay good money for my 12 pro and if the new iPhone has finger print I will get it. I think all of us would prefer that apple focus on addressing the bugs instead of nice features.

Submitted by Khushi on Friday, May 21, 2021

I'm honestly excited for voiceover features but not sure whether my IPhone 6s would be getting the benifits of these features.
its highly unlikely according to me though because I don't think I'll be receiving IOS 15. don't know anything though.

Submitted by Andy Lane on Friday, May 21, 2021

Its so sad its got to the point that I’m going to go out on a limb and say I’m guessing I’m not the only one who at times really hates their Apple device. It shouldn’t be like this. I’m pretty sure they didn’t intend it to be like this and before anyone jumps on me for saying anything negative, I recognise how much they have done for mainstream accessibility while at the same time recognising how much they are doing to make my experience often frustrating, sometimes maddening and occasionally make the fact I can’t see so so very much worse than it needs to be and limits me in ways I hadn’t realised until I really thought about it. Just as an example, does anyone else order EVERYTHING from Amazon simply because their app has the best accessibility and filling in websites, working around adds pop ups and inaccessible baskets and payment screens? Thats not to say Amazon have a consistently good UX just that its mostly less frustrating. I still can’t move through search results easily, sometimes can’t select number of items to buy, lose focus on when expanding a review and on and on. Its just better than dealing with all that rubbish and much much more on the web. Thats so limiting, It limits my choice just because I can’t be doing with working around frustrations that don’t need to be there. I have literally spent 3 hours trying to buy something once on a website. Thats not good enough and I really hated my iPhone after that which doesn’t need to happen if bugs were fixed and accessibility issues given parity with mainstream. The fact I can’t edit a text document consistently should be number 1 on someone’s list of bugs to fix. It would be if it affected mainstream. The fact my phone regularly screams in my ear when its connected to a Bluetooth device Makes me so upset and physically hurt sometimes that I hate apple, hate I have to use such a shoddy solution to a problem in life and hate that I’m a second class citizen when it comes to resolving the issues on a device I spent £1250 on. Its not good enough and someone should be staying up late to stop it happening. If this was happening in the mainstream, it would be all over the media much bigger than antennagate, but I still have the problem 2 years or more after it was introduced. Remember the infrared sensor for proximity in calls? how often has that failed and blasted your ear with full speaker volume? i knew it was going to happen when they put the speaker in the ear peace and surprise surprise it did. I know I sound frustrated, I am. Can you say you’ve never been incredibly frustrated with your device? For anyone saying yeh but remember iOS 13, its not as bad as that. Thats not a defence, thats the problem. And yet we keep buying the devices because they are for all their frustrations windows to the world, they’re great in so many ways but are let down by things that could be fixed so easily by a thorough and robust approach to squash resolving bugs, giving them their correct priority and understanding how they impact peoples lives. If a website lost focus every time you use an element for someone sighted that would be quite annoying, they’d have to flick down each time but for someone blind it could take 30 seconds to get back to an element on some pages for that only to happen again and again? Its a very high priority issue that needs resolving. Anyway, I think I’ve gone on enough. If anyone agrees with me that this isn’t good enough and needs to change, do you have a workable plan or any ideas how to make apple aware of the impact these issues have on our lives? I’m sure if they understood the frustration their bugs cause they’d want to do more to put them right. I refuse to believe they only work on accessibility for the funky videos and ads. Its not that, they really do care but don’t understand how bad things are for some of their users. Like those who use Mac for example. I’m not one of them anymore as I couldn’t cope with the bugs.

Submitted by DMNagel on Friday, May 21, 2021

There is just no way that everyone can agree. Many of us may have the same devices, but we don't always use it for the same stuff. Therefore, it is impossible for everyone to be on board when it comes to certain problems. I cannot for example, complain about problems I don't even have. I may even have them, but don't realise it since I haven't placed myself in a position to bump into them. Perhaps the greatest example is the fact that some people are griping about VoiceOver bugs that are language specific, so there is no way we can or even will agree with you on those, since we don't even touch it ourselves. Not that I don't want peoples problems to be solved mind you, but whatever I'm using my device for seems to be working fine, so don't expect me, or anyone like me to go trashing and bashing apple alongside you. Perhaps our day will come sooner or later. When it does, you will hear from us. Until then, I'll be looking forward to having some new stuff.

Submitted by Lielle ben simon on Friday, May 21, 2021

Club AppleVis Member

In hebrew the Braille" table isn't fixed from iOS 9, and it getting wars from version to version. There is liblouise table, but I can't to write in hebrew, to read only. I hope that it will fix on iOS14, for everyone.

Submitted by Dennis Long on Friday, May 21, 2021

To the person that has issue with amazon.com website. I can't say I have issue. Granted I use a physical keyboard however I don't think that makes a huge difference. Between iOS and android iOS is most definetly the more polished system.

Submitted by PaulMartz on Friday, May 21, 2021

Member of the AppleVis Blog Team

Allowing OCRs of receipts to be navigated as tables reminds me that we also need OCR enhancements to improve navigation of restaurant menus. If I OCR a page of a restaurant menu, my device should identify section headings by differences in font or text size and allow me to navigate by those headings, so I don't have to swipe through a dozen salads that don't interest me just to get to the soups, for example.

Of course, if the restaurant had an accessible online menu, that would be an even better solution, avoiding the clumsy OCR altogether. But I have yet to see a restaurant with an accessible online menu, and I'm not going to hold my breath waiting for every restaurant to make such a change.

Submitted by Blee Blat on Friday, May 21, 2021

I'm assuming that you'd need something newer than an iphone 8 for these features? I'm trying to work out when's the best time to upgrade my device. The problem is the newer devices don't give me anything I really care about, unless someone can convince me that these swank cameras actually make things more accessible. But I suppose that's possibly another forum thread.