Does the Apple Pencil become a viable tool for the blind in IOS14 with character recognition?

Also this new Scribble feature where you can draw shapes too. If the pencil is accessible for the blind then I would like to write in cursive again. I loved writing too!

I'd be surprised if there wasn't some accessibility to it. It seems like a great feature for someone with fluctuating sight, or even some residual. Like brailing, it only relies on muscle memory rather than seeking out buttons on a sheet of glass. Could be a powerful input tool.

I also would LOVE to be able to quickly handwrite notes and be able to read them with VO. I used to be fully sighted and still have some usable vision, but it isn’t very reliable. I’ve been playing around with the Apple Pencil with iPad OS 14.4 (on iPad Air 4th generation). I wouldn’t recommend anyone who is completely blind to buy it yet since it doesn’t really work with VO, but the handwriting-to-text is phenomenal so I think it has amazing potential. To be clear, I am not referring to the handwriting character input method, which isn’t accurate enough to be useful (in my experience).

For more specifics of what I’ve experienced with the Apple Pencil, read on...

There is currently no way to input text with the Apple Pencil with VO turned on. There is only one exception: you can double tap and hold with the Apple Pencil, wait for the sound that indicates the pass-through has begun, and then scribble something without lifting the pencil... but as soon as you lift the pencil, it’s over... so it’s not useful. Alternatively if you have some vision to work with you can toggle off VO. The handwriting-to-text conversion in real time is extremely accurate. However, I can’t seem to predict where the text will actually appear on the screen. Maybe it has to do with my large text visual settings or perhaps it’s just that my vision can’t be trusted, but I can’t seem to use it this way. Plus without the VO feedback it’s hard to know if it’s inputting correctly or is a bunch of jibberish with text inserting in the middle of a sentence, etc.

Something else I’m playing with now is to turn VO off and create a purely handwritten note (which FYI VO refers to later as a “handwriting attachment” and can read in full really well). It is possible to select all the handwriting and then “copy as text.” I’m having a lot of trouble with this step since VO needs to be turned off, but after that I can just toggle on VO and paste the text above the handwriting attachment. And I leave the handwriting attachment there so if need be a sighted person can make sense of it for me later. I’m not sure how useful this will prove to be, but the Apple Pencil input has great potential since: (1) the accuracy is truly amazing, and (2) it’s super easy to create a new handwritten note with the Apple Pencil by just tapping the Pencil on the iPad when the screen is off. Again, VO needs to be off to input though.

If there are any other tricks or tips out there, I’d love to hear them!!! ...especially if there is a way to select all the handwriting and copy as text with VO turned on. Thanks!

P.S. I’d also take more general tips on using the Notes app with VO since it doesn’t seem to behave like other text editors... Perhaps I am missing something obvious...?

Here’s an idea on a user interface design that could make the Pencil useful while VoiceOver is turned on. I’m not an Apple developer and I probably don’t even have all the terminology right, but if it were possible to implement it, I think it would be a game changer... For now it’s just my musings but I wonder if anyone else has thought about this ...

Firstly, given that:
1. the Apple Pencil already is enabled with a double-tap gesture.
2. VO has the ability to “pass-through” a standard gesture (triggered by a double-tap and hold) but it’s short-lived.
3. It is already possible to allow handwriting/drawing input from only the Pencil, so if you accidentally touch the screen while drawing, your hand won’t make a mess of it.

What if... the VO pass-through could be toggled on/off using a customized gesture on the Apple Pencil itself? For example, after navigating the VO cursor to a text input area (e.g. a container that allows text/handwriting input such as Notes or Mail), a double tap (or even triple tap!?) on the Pencil would lock the VO cursor on that input area and also lock the pass-through on to allow handwriting input from the Pencil. Any input from the Pencil at that point would be interpreted as handwriting, with the possibility of the handwriting-to-text input method into that area where the VO cursor is locked. In that case of handwriting-to-text, VO would speak aloud the text as it is converted, just like it speaks words as they are entered when typing. This state could then be toggled off with the same gesture on the Pencil and normal VO input would resume.

Hi jdubz, this might be new with iOS 15 but you can start handwriting with VO on and continue writing new words. I think if you pause for an extended period of time then it will exit Scribble mode. It's still clunky and leaves much to be desired though.

I've been using it with magnification which has its own quirks but I'm hoping to figure out the best way to use it

I would love to know more details about how you are able to make this work if you are successful.

It’s really just like you said. I enter the field, double-tap and hold the Apple Pencil until the sound occurs, then I write normally. I’m able to lift it up from the iPad screen. Then when I stop writing and wait a few seconds another sound occurs to signal that it’s stopped.

Having said that, it’s hardly usable and so prone to error. I’m guessing it has something to do with how the VO screen operates. For example, the point where the pen meets the screen is not where the writing is displayed. It’s usually at the top of the text so that your cursor starts getting mixed into what’s being typed. Words and characters can easily become jumbled.

I'd urge you, and anyone who is interested in this mode of input, to contact

accessibility@apple.com

to tell them your thoughts. They are looking into how the apple pencil could work better for those using VoiceOver and would like use case scenarios with the device. Perhaps tell them how you'd foresee it working a useful way, eg not having to write in the correct place and accepting it just continues on much like the brail keyboard input... Just an example. I'm borrowing an Apple Pencil later today to have a play with it and get down some ideas.

I think it's just another tool that could be used in creative ways for us to access the iPad, EG wand movements, tap controls.