[Work In Progress]: Vosh - a third-party screen-reader for the Macintosh

If you mean when you expand a selection using the text caret, yes, it can be read.

If you mean when you move the screen-reader cursor to the element, there are several pieces of optional information that can be present and can be read too.

If you mean reading the element under the mouse cursor, yes, there's a way to do that too.

The project is currently stalled though, because I've been trying to reverse engineer VoiceOver to extract undocumented and undeclared notification identifiers and figure out how to make Vosh notice changes to the frontmost window without polling the system-wide element. Also the video linked to on the original post no longer reflects the state of the project.

I've abandoned this project.

Originally I wanted to build something that could showcase my ability to develop for Apple technologies, with the intent of landing a job as a native iOS developer. However since nobody replies to my applications, I have switched to a much more interesting yet niche area and language: embedded / bare metal development in Rust. Unfortunately while I have a lot of free time on my hands, I really do need a job, so currently that's the highest priority in my life.

The project's repository will remain public just in case someone wishes to fork or learn something from it, but I'm unlikely to pick it up again in the near future.

I believe it's possible to use VoiceOver with just one hand, thanks to QuickNav, Single Key QuickNav, and VO locking. To enable QuickNav, press the Left and Right Arrows simultaneously by default; to enable Single Key QuickNav, press VO+Q by default; to enable VO locking, press VO+Semicolon. Unfortunately if VO locking is designed for single handed users, the position of the keys at least on US laptop keyboards makes it very inaccessible since there's no Control key anywhere near the Semicolon. It can also be activated from the VoiceOver Help Menu (VO+H) -> General -> Toggle the VO Modifier Lock On or Off, and might be callable from Keyboard Commander possibly using AppleScript so you might be able to map it to a more accessible key combination (I didn't test this).

For starters, I've learned a great deal about Apple's accessibility infrastructure since I abandoned Vosh almost two years ago, and I am aware of memory safety issues resulting from Apple's bad API design which enable race conditions in concurrent code, as that code was abused to the limit in my previous job and exposed some problems that I ended up working around since I can't really fix the Apple implementations that I depend on everyone else's computers. I can provide the same kind of fix to Vosh if needed, just need to know exactly where I should make the related pull requests.

There's also one feature that I did not support in my safe abstraction which are custom rotors since there's no public API for that so I'll definitely have to do some reverse-engineering to figure out how that feature works at the lowest levels and how to interact with it so that I can provide yet another safe abstraction. The work I did at my former job aimed towards leveraging the accessibility and audio infrastructures to deliver a highly integrated AI agent on macOS, so only a subset of features was really required, and custom rotors were definitely not in that feature subset.

As for iOS support, that is unfortunately not possible right now even here in the European Union with the Digital Markets Act cracking that platform open. I could probably file a complaint about that to get Apple to provide third-party developers with the ability to interact with the accessibility infrastructure from the consumer-side, but won't do that since I don't think there's a need right now, I don't plan on working on that any time soon if ever, and I don't engage in or trigger frivolous legal battles.

As for the code, my version of it which hasn't been change for nearly 2 years and which I have since archived, is still available on my GitHub profile. I can probably make a video demo of it, but since it has been forked and improved by another developer, I don't think it would be very useful, so I'll probably wait to learn about where the main repository is now and then maybe make a demo with the improvements already in place.

Finally I honestly no longer approve of my own accessibility API design, not because it's flawed or anything, but I have since figured out much better ways to abstract away Apple's lower level APIs taking advantage of modern Swift features like result builders, property wrappers, and observers, and providing a new Swift package with a better designed API is a personal project with a low priority in my pipeline. I also want to build a proper machine learning solution based on my own analytic geometry ideas that can then be trained to provide a very robust and highly optimized optical character recognizer running on the GPU for macOS and iOS / iPadOS, which is a project with a very high priority for me that I will tackle immediately after finishing the cloud software infrastructure for my upcoming business which I will also take advantage of for personal stuff, and could definitely be integrated in any third-party screen-reader to augment the experience and attempt to provide accessibility even in totally inaccessible applications.

Many problems that occur with VoiceOVer come from bad design decisions in the underyling Accessibility API and bad implementations in Cocoa (NSTextView) and other places.

I think that most of Cocoa is fine, it's just NSTextView that is kind of a show stopper because it's simply not providing proper accessibility information. The infrastructure is also fine for the most part, it's just the consumer-side Carbon API that is extremely poorly designed and implemented, especially when it comes to concurrency, but this can be worked around or even reimplemented, and there's a huge lack of documentation and even identifier declarations in the public APIs, which in some cases can easily be learned by simply dumping accessibility trees whereas in other cases hardcore reverse-engineering is required to figure out how things are actually implemented at the lowest levels. There are also some system services that are bad accessibility citizens by announcing that they do implement accessibility but never really responding to accessibility queries, requiring keeping track of their bundle identifiers in order to avoid ever interacting with them.

Also, the web is even more complex as the rendering engine creates its own accessibility tree that you'd have to "translate" into native UI elements or something else that implements the Accessibility protocol.

If you look into WebKit, for example, and their accessibility object wrapper, it's a pure mess to make functionality like text navigation, reading by line, character, or word, possible at all. You might overcome this with massive effort of trial and error and a lot of digging into the depth of Mach traffic that is exchanged to get details that the documentation is not telling you (And this is alot!), but as the accessibility API is that poor at the moment, I'd say that this would be waste of time.

Those are provider-side problems. From the consumer-side, which is where screen-readers and other accessibility services operate, both WebKit and Blink provide properly abstracted accessibility trees not too different from the ones provided by Cocoa. The fundamental problem when it comes to web navigation is that VoiceOver itself is quite broken, and fixing it wouldn't even be that hard. The problems with caret browsing are real and may stem from a deeper issue related to how attributed strings are communicated to the accessibility infrastructure that I haven't investigated yet, but my empirical observation, which I can elaborate further on if necessary, tells me that caret browsing is implemented in VoiceOver itself, not WebKit, meaning that it can be reimplemented properly without messing with WebKit.

My take on this is that alone it is indeed a very ambitious project, but with a community it can be tackled just like in NVDA, so if providing people with the safe abstractions is all that's needed for other people to engage, I can dedicate some time to that effort. However back in 2023 only one person engaged with the project to any capacity, and my life has changed a lot since those days, when I had too much time on my hands and very little to do, as these days it's the exact opposite, so I can no longer dedicate all my time to this kind of niche project.

After reviewing my own code, my conclusion is that it's not as bad as I thought. I mean the lower level abstractions are just as bad as I remember them since that's the module that I based my professional work on, but I did implement some abstractions on top of that that even two years later I still consider properly designed.

The Swift package has 5 modules, with the Vosh module being the executable target that serves as the front-end to everything else, the Access module being a library target with the actual screen-reader which orchestrates everything, the Input module being a library target handling input devices (currently only keyboards and keypads), the Output module being a library target handling semantic audio output (currently only speech), and the Element module being a library target providing a safe abstraction directly on top of Apple's Carbon accessibility API which is what I was considering redesigning. In the Access module, there's a type called AccessReader which provides semantic reading functionality. This module delegates its implementation to AccessGenericReader and its subclasses that implement the reading strategies for all kinds of element types. These strategies are far from complete but are designed to be extensible, so all that's needed to add more strategies is to create new subclasses and customize the reading strategy for a specific element role by overriding the methods corresponding to the semantic aspects that need customization.

I will implement the workarounds for the known bugs in Apple's Carbon accessibility API, and fix a bug in my own accessibility actor that is likely making its dedicated thread exit unexpectedly when no accessibility observers are alive, which will be two separate contributions in their respective commits that I will then suggest as a pull request to the repository above. I still intend to figure out how to consume custom rotors at some point, and will look into that as well, but unless I end up finding out that the task is not very daunting, I will postpone that to a later stage depending on community engagement. Lastly I will review and test the modified code on the other repository before deciding on whether I should synchronize my main branch, and maybe publish some audio or video with a demo, where understanding what I'm saying should be a lot easier than in the original video since working for Americans forced me to speak English a lot more frequently so my diction and pronunciation have improved significantly since 2023.

I started updating code yesterday, first I checked Ashley's code, which has over 11000 lines of difference from mine, and then decided to go back to my own version of the code, and update the entire codebase to macOS 26 and Swift 6.2 with full strict concurrency checking along with addressing the problems that I was already aware of, which is why this is taking a while to finish.

Beyond the sheer amount of changes made by Ashley, I do have issues with most if not all of the changes themselves, some of which introducing unsafe AI slop regressions, as even the README file was not spared from a drunken robot rewrite complete with the removal of my safety warnings. AI slop is already bad in itself because current models have a tendency to not really understand and definitely not produce properly designed abstractions, on top of introducing very subtle bugs that require a lot of reviewer skill to spot even in relatively confined updates, but is especially reckless in this case if we consider the security implications resulting from the privileges required to run a screen-reader that takes full control over the system. Therefore if his changes were suggested in a pull request to my repository, I would have no doubt about completely discarding them as utter trash.

While I have no issue with inexperienced software engineers and have been actively sharing knowledge with other people for well over 2 decades, which is almost as long as I've been coding myself, I strongly oppose any kind of vibe coding, because current AI models are not designed to learn from their mistakes, and because even if they did, that would still not translate to other humans actually learning anything. Therefore to me the resources spent producing this kind of code, and the time spent reviewing it, are totally worthless and thus vibe coding is a concept that I will never accept even in prototyping regardless of how productive generating code that way may appear. I got people fired from the engineering team at my former job because of their tendency to vibe code, as a demonstration of how strongly I oppose vibe coding and how seriously I take my stance in this regard.

Having noticed the above, I have grown especially concerned about the safety of established open-source screen-readers like NVDA and Orca, because while technical skill was a somewhat effective barrier to entry before ChatGPT entered the playing field in 2022, the fact that anyone can pretend to write code these days means that there's likely already a huge amount of bad quality AI slop running in critical production services with potentially huge security implications down the line, so I cannot stress my repudiation of vibe coding hard enough.