Does the Apple Pencil become a viable tool for the blind in IOS14 with character recognition?

Forum
Hardware & Accessories

Hi,

I was wondering what people's thoughtts were on the apple pencil and IOS14. My understanding is that it can be used to write text that is then converted into digital text, similar to the handwriting feature in universal access.

Firstly I was wondering if this feature is accessible and secondly, whether it was something that could be useful.

I'm a writer, or try to be, and there is something exciting about the slow thoughtful method of writing down ideas long hand rather than blasting them out at 100 words per minute. Maybe it's just a call back to when I could see and write, but there is a certain romance to hand writing something even if the output is a digital content.

Options

Comments

Submitted by Blind angel 444 on Thursday, July 23, 2020

Also this new Scribble feature where you can draw shapes too. If the pencil is accessible for the blind then I would like to write in cursive again. I loved writing too!

Submitted by Oliver Kennett on Thursday, July 23, 2020

I'd be surprised if there wasn't some accessibility to it. It seems like a great feature for someone with fluctuating sight, or even some residual. Like brailing, it only relies on muscle memory rather than seeking out buttons on a sheet of glass. Could be a powerful input tool.

Submitted by JDubz on Wednesday, March 3, 2021

I also would LOVE to be able to quickly handwrite notes and be able to read them with VO. I used to be fully sighted and still have some usable vision, but it isn’t very reliable. I’ve been playing around with the Apple Pencil with iPad OS 14.4 (on iPad Air 4th generation). I wouldn’t recommend anyone who is completely blind to buy it yet since it doesn’t really work with VO, but the handwriting-to-text is phenomenal so I think it has amazing potential. To be clear, I am not referring to the handwriting character input method, which isn’t accurate enough to be useful (in my experience).

For more specifics of what I’ve experienced with the Apple Pencil, read on...

There is currently no way to input text with the Apple Pencil with VO turned on. There is only one exception: you can double tap and hold with the Apple Pencil, wait for the sound that indicates the pass-through has begun, and then scribble something without lifting the pencil... but as soon as you lift the pencil, it’s over... so it’s not useful. Alternatively if you have some vision to work with you can toggle off VO. The handwriting-to-text conversion in real time is extremely accurate. However, I can’t seem to predict where the text will actually appear on the screen. Maybe it has to do with my large text visual settings or perhaps it’s just that my vision can’t be trusted, but I can’t seem to use it this way. Plus without the VO feedback it’s hard to know if it’s inputting correctly or is a bunch of jibberish with text inserting in the middle of a sentence, etc.

Something else I’m playing with now is to turn VO off and create a purely handwritten note (which FYI VO refers to later as a “handwriting attachment” and can read in full really well). It is possible to select all the handwriting and then “copy as text.” I’m having a lot of trouble with this step since VO needs to be turned off, but after that I can just toggle on VO and paste the text above the handwriting attachment. And I leave the handwriting attachment there so if need be a sighted person can make sense of it for me later. I’m not sure how useful this will prove to be, but the Apple Pencil input has great potential since: (1) the accuracy is truly amazing, and (2) it’s super easy to create a new handwritten note with the Apple Pencil by just tapping the Pencil on the iPad when the screen is off. Again, VO needs to be off to input though.

If there are any other tricks or tips out there, I’d love to hear them!!! ...especially if there is a way to select all the handwriting and copy as text with VO turned on. Thanks!

P.S. I’d also take more general tips on using the Notes app with VO since it doesn’t seem to behave like other text editors... Perhaps I am missing something obvious...?

Submitted by JDubz on Tuesday, April 6, 2021

Here’s an idea on a user interface design that could make the Pencil useful while VoiceOver is turned on. I’m not an Apple developer and I probably don’t even have all the terminology right, but if it were possible to implement it, I think it would be a game changer... For now it’s just my musings but I wonder if anyone else has thought about this ...

Firstly, given that:
1. the Apple Pencil already is enabled with a double-tap gesture.
2. VO has the ability to “pass-through” a standard gesture (triggered by a double-tap and hold) but it’s short-lived.
3. It is already possible to allow handwriting/drawing input from only the Pencil, so if you accidentally touch the screen while drawing, your hand won’t make a mess of it.

What if... the VO pass-through could be toggled on/off using a customized gesture on the Apple Pencil itself? For example, after navigating the VO cursor to a text input area (e.g. a container that allows text/handwriting input such as Notes or Mail), a double tap (or even triple tap!?) on the Pencil would lock the VO cursor on that input area and also lock the pass-through on to allow handwriting input from the Pencil. Any input from the Pencil at that point would be interpreted as handwriting, with the possibility of the handwriting-to-text input method into that area where the VO cursor is locked. In that case of handwriting-to-text, VO would speak aloud the text as it is converted, just like it speaks words as they are entered when typing. This state could then be toggled off with the same gesture on the Pencil and normal VO input would resume.

Submitted by Ekim on Tuesday, October 5, 2021

Hi jdubz, this might be new with iOS 15 but you can start handwriting with VO on and continue writing new words. I think if you pause for an extended period of time then it will exit Scribble mode. It's still clunky and leaves much to be desired though.

I've been using it with magnification which has its own quirks but I'm hoping to figure out the best way to use it

It’s really just like you said. I enter the field, double-tap and hold the Apple Pencil until the sound occurs, then I write normally. I’m able to lift it up from the iPad screen. Then when I stop writing and wait a few seconds another sound occurs to signal that it’s stopped.

Having said that, it’s hardly usable and so prone to error. I’m guessing it has something to do with how the VO screen operates. For example, the point where the pen meets the screen is not where the writing is displayed. It’s usually at the top of the text so that your cursor starts getting mixed into what’s being typed. Words and characters can easily become jumbled.

Submitted by Oliver Kennett on Wednesday, October 6, 2021

I'd urge you, and anyone who is interested in this mode of input, to contact

accessibility@apple.com

to tell them your thoughts. They are looking into how the apple pencil could work better for those using VoiceOver and would like use case scenarios with the device. Perhaps tell them how you'd foresee it working a useful way, eg not having to write in the correct place and accepting it just continues on much like the brail keyboard input... Just an example. I'm borrowing an Apple Pencil later today to have a play with it and get down some ideas.

I think it's just another tool that could be used in creative ways for us to access the iPad, EG wand movements, tap controls.