Does the Apple Pencil become a viable tool for the blind in IOS14 with character recognition?

Submitted by Oliver Kennett on Thursday, July 23, 2020.
Forum
Hardware & Accessories

Hi,

I was wondering what people's thoughtts were on the apple pencil and IOS14. My understanding is that it can be used to write text that is then converted into digital text, similar to the handwriting feature in universal access.

Firstly I was wondering if this feature is accessible and secondly, whether it was something that could be useful.

I'm a writer, or try to be, and there is something exciting about the slow thoughtful method of writing down ideas long hand rather than blasting them out at 100 words per minute. Maybe it's just a call back to when I could see and write, but there is a certain romance to hand writing something even if the output is a digital content.

Options

Comments

Submitted by Blind angel 444 on Thursday, July 23, 2020

Also this new Scribble feature where you can draw shapes too. If the pencil is accessible for the blind then I would like to write in cursive again. I loved writing too!

Submitted by Oliver Kennett on Thursday, July 23, 2020

I'd be surprised if there wasn't some accessibility to it. It seems like a great feature for someone with fluctuating sight, or even some residual. Like brailing, it only relies on muscle memory rather than seeking out buttons on a sheet of glass. Could be a powerful input tool.

Submitted by JDubz on Wednesday, March 3, 2021

I also would LOVE to be able to quickly handwrite notes and be able to read them with VO. I used to be fully sighted and still have some usable vision, but it isn’t very reliable. I’ve been playing around with the Apple Pencil with iPad OS 14.4 (on iPad Air 4th generation). I wouldn’t recommend anyone who is completely blind to buy it yet since it doesn’t really work with VO, but the handwriting-to-text is phenomenal so I think it has amazing potential. To be clear, I am not referring to the handwriting character input method, which isn’t accurate enough to be useful (in my experience).

For more specifics of what I’ve experienced with the Apple Pencil, read on...

There is currently no way to input text with the Apple Pencil with VO turned on. There is only one exception: you can double tap and hold with the Apple Pencil, wait for the sound that indicates the pass-through has begun, and then scribble something without lifting the pencil... but as soon as you lift the pencil, it’s over... so it’s not useful. Alternatively if you have some vision to work with you can toggle off VO. The handwriting-to-text conversion in real time is extremely accurate. However, I can’t seem to predict where the text will actually appear on the screen. Maybe it has to do with my large text visual settings or perhaps it’s just that my vision can’t be trusted, but I can’t seem to use it this way. Plus without the VO feedback it’s hard to know if it’s inputting correctly or is a bunch of jibberish with text inserting in the middle of a sentence, etc.

Something else I’m playing with now is to turn VO off and create a purely handwritten note (which FYI VO refers to later as a “handwriting attachment” and can read in full really well). It is possible to select all the handwriting and then “copy as text.” I’m having a lot of trouble with this step since VO needs to be turned off, but after that I can just toggle on VO and paste the text above the handwriting attachment. And I leave the handwriting attachment there so if need be a sighted person can make sense of it for me later. I’m not sure how useful this will prove to be, but the Apple Pencil input has great potential since: (1) the accuracy is truly amazing, and (2) it’s super easy to create a new handwritten note with the Apple Pencil by just tapping the Pencil on the iPad when the screen is off. Again, VO needs to be off to input though.

If there are any other tricks or tips out there, I’d love to hear them!!! ...especially if there is a way to select all the handwriting and copy as text with VO turned on. Thanks!

P.S. I’d also take more general tips on using the Notes app with VO since it doesn’t seem to behave like other text editors... Perhaps I am missing something obvious...?

Submitted by JDubz on Tuesday, April 6, 2021

Here’s an idea on a user interface design that could make the Pencil useful while VoiceOver is turned on. I’m not an Apple developer and I probably don’t even have all the terminology right, but if it were possible to implement it, I think it would be a game changer... For now it’s just my musings but I wonder if anyone else has thought about this ...

Firstly, given that:
1. the Apple Pencil already is enabled with a double-tap gesture.
2. VO has the ability to “pass-through” a standard gesture (triggered by a double-tap and hold) but it’s short-lived.
3. It is already possible to allow handwriting/drawing input from only the Pencil, so if you accidentally touch the screen while drawing, your hand won’t make a mess of it.

What if... the VO pass-through could be toggled on/off using a customized gesture on the Apple Pencil itself? For example, after navigating the VO cursor to a text input area (e.g. a container that allows text/handwriting input such as Notes or Mail), a double tap (or even triple tap!?) on the Pencil would lock the VO cursor on that input area and also lock the pass-through on to allow handwriting input from the Pencil. Any input from the Pencil at that point would be interpreted as handwriting, with the possibility of the handwriting-to-text input method into that area where the VO cursor is locked. In that case of handwriting-to-text, VO would speak aloud the text as it is converted, just like it speaks words as they are entered when typing. This state could then be toggled off with the same gesture on the Pencil and normal VO input would resume.