Designing a new equation-editing app - what should I focus on?

Forum
App Development and Programming

I'm looking for advice from VoiceOver users about what they would expect from an equation-editing app.

I am an app developer designing an equation-editing app for iPhone, iPad and Mac. Think Microsoft word but for creating and editing equations. It will be an app rather than a plug-in. However, it will be easy to share equations to include them in documents in other apps like Microsoft Word or export them as LaTex. My new app will not solve equations but can be used to create and edit equations. I have made one previous VoiceOver-optimised app so I have some experience.

What should I be concentrating on to ensure easy use with VoiceOver in an equation-editing app?

Options

Comments

Submitted by Kevin Shaw on Thursday, September 30, 2021

Thanks for taking a stab at this. I've given this problem some careful thought over the past few years.

One of the big drawbacks to reading in-line ASCII equations is not hearing how the equation is supposed to sound when read aloud.

e = MC^2 should be read as e equals mc squared. I would love the ability to move past each of the coefficients and know where in the equation it is relative to the other coefficients. This is an easy one as it all fits on one line. C squared is its own coefficient and a fairly easy one to grasp.

For more complex equations, we start running into issues:

V = f/ᴧ: read aloud as V equals f over lambda. Is there a way to communicate the spatial relationship of f over lambda relative to V?

Another one:

ps = T-D/W*V: Read aloud as: P sub s equals T minus D over W times V. (This is a formula for measuring energy states in jet fighters.)

Question: In this formula, is the quantity T minus D to be divided by W or, is D over W to be subtracted from T?

I have found it helpful to use the term, "the quantity" to describe what is in brackets. For example, the quantity X squared minus 1 times the quantity X squared plus 1. This lets me know the terms of the quantity are self-contained. Inspecting by character could reveal brackets or parens.

Where T minus D is all over W, I would like to hear VoiceOver say something like: "P sub s equals the quantity T minus D over W……. In the case where the term "D over W" is to be subtracted from T, VoiceOver would have to distinguish this by saying, "V equals T minus the fraction D over W…".
The next part of the equation becomes tricky. If VoiceOver says, "Times V, my expectation is that the fraction T minus D over W is to be multiplied by V. If I hear divided by WV, I would know W is to be multiplied by V and that WV is its own term.

VoiceOver should be able to handle exponents with the terms like, "to the power of ~" and subscripts with the term "sub ~".

There is a part of this program that would definitely benefit from having someone who knows the linguistics of math or science to develop the vocabulary which would dictate how VoiceOver would speak the equation. If this is laid out in a concise way, there could be some type of NLP interface that would allow someone to dictate the formula.

I would also look at using the Apple Pencil to write in equations and have them translated to a usable form on iPad. Some of us with sight loss know how to write out the equation. Perhaps there is an affordance to do this in a stepwise fashion.

It would also be helpful for those who know Nemeth braille code to key formulae in with refreshable braille displays.

Always happy to add more thoughts.

Submitted by peter on Thursday, September 30, 2021

First, as you probably know, blind people would have difficulty reading equations and expressions that appear in the 2D format that sighted people are used to. Any equations or expressions I have worked with using speech or braile are format in a 1D, linear format.

Secondly, it can be difficult for a blind person to understand the logical relationships in long and/or complex expressions. One method of coping with this is to have a gesture or keyboard shortcut for easily moving from, say, a left parenthesis to the corresponding right parentheses and vice versa.

Probably the best way of representing complex expressions to the blind is how most screen readers are dealing with MathML expressions these days. using mathML and a screen reader one can navigate from sub expression to sub expression, up and down an expression at a time, or navigate within an expression. In this way logicaly connected content is kept together and the blind person can have a better understanding of the overview of the expression.

I have also made use of assigning variables to sub expressions. For example a complex expression might have many sub expressions within it. If the larger expression is written in terms of numberbed variables, the user can then query each of the numbered variables separately to see the details of each sub expression but yet still have a good overview of the larger expression and how it is put together.

Hope that feedback helps. As a retired research physicist who is blind, I know that blind people can have rewarding careers in STEM and like to see work such as yours that makes tools like this readily available to those considering careers in STEM.

Keep up the good work and keep us posted on progress.

--Pete
P.S. If you are eventually interested in adding the capability of actually solving equations and performing symbolic math, look into some of the CAS open source projects that could make this possible. Maxima is a great example that works on PC's and I believe people might have worked on porting such tools to mobile devices.

Submitted by Jenna Pepper on Thursday, September 30, 2021

Club AppleVis Member

I've no feedback to add. What's here is pretty good, but I'd like to say, as a student who uses Notepad++, 1D format, a Braille display, lots of profanity, and legal depressants to write her equations, that I will pay lots of very much money for an app that helps make my math courses less of a headache. I wait excitedly for what you make.

Thank you Kevin,

I will add the following features to my list:
1. Develop a vocabulary which dictates how VoiceOver speaks the equation. VoiceOver should read equations how they are supposed to sound when read aloud. There should be some clarifying terms, for example 'the quantity' to describe what is in brackets. VoiceOver should be able to handle exponents with terms like, "to the power of ~" and subscripts with the term "sub ~".
2. Allow a user to dictate an equation, possibly using an NLP interface.
3. Allow use of the Apple Pencil to write in equations and have them translated to a usable form. Ideally, this would be possible in a stepwise fashion.
4. Allow formulae input by Nemeth braille code from refreshable braille displays.

I can't promise all the features will make it into version one. But, I'll do my best.

What do you think about using the clarifying terms fraction, numerator, denominator and end of fraction? The jet fighter energy states equation would then be read aloud as: P sub S equals fraction numerator T minus D denominator W end fraction times V. It might be clearer. But, less concise.

Thank you Pete,

What do you think about having two options?
Option A: The equation is displayed in 2D format
Option B: The equation is displayed in 1D linear MathML format
There will be a switch in settings to change between options.

In 2D format, VoiceOver will read the whole equation correctly when asked. And a user will be able to find different parts of the equation by moving their finger over the screen on iPhone and iPad. Tapping on the numerator will read out the numerator for example. And allow editing of just that numerator. Every equation will be treated as a collection of cells (sub-expressions) such as numerators and denominators that can be individually read and edited. As default, VoiceOver will read the whole equation line in one. But, invoking a rotor action can get VoiceOver to step through the equation cell by cell. The user can stop at the cell they want to edit.

Similarly, in 1D format, I will take the approach you say has worked in other apps. A user can navigate from sub expression to sub expression, up and down an expression at a time, or navigate within an expression. By default, VoiceOver will read the MathML as written. But, invoking a rotor action will get VoiceOver to read the expression as it would for the 2D format.

Other features to add to my list are:
1. A gesture or keyboard shortcut for easily moving from, say, a left parenthesis to the corresponding right parentheses and vice versa.
2. Assign each cell (sub expression) a numbered variable so the user can query each of the numbered variables separately eg LC1 for the first sub expression on the left hand side of an equation. Or just CELL1 if the expression has no equality-type symbol separating left and right sides.
3. Allow solving equations and performing symbolic math, possibly using CAS open source projects. See maxima.sourceforge.io for example. (This feature will likely be at a later stage).

Submitted by CJ on Friday, October 1, 2021

In reply to by Jenna Pepper

Thank you Jenna,

I was already feeling pretty motivated. But, your post will help keep me going when the coding gets difficult!

CJ,

Your project sounds pretty ambitious and it could be a great resource for the blind.

One concern I have is about your impressions of how a blind person would use Voiceover in this app or when navigating mathematical expressions.

For example, since you are sighted, you presumably think of touching a particular area that might correspond to, for example, the numerator or denominator of an expression. This is likely not the way that a blind user would expect to find sub expressions. common gestures for navigating with Voiceover are to flick right/left or flick up/down. For example if I were moving from paragraph to paragraph in a printed article I wouldn't move my finger down the page. Rather I would flick to the next heading, line, character, etc. depending on how I set the rotor and how I wanted to navigate. I don't think a 2D representation would be useful for someone who is blind. On the other hand, the 2D representation would be useful when a blind person is trying to share their work with a sighted person, so the ability to switch back and forth would be of some use.

Anyway, when you develop this app, be sure to have blind people actually use it and provide feedback on their particular work styles (which might be very different than how you might interact with the UI!). You might even try turning on Voiceover, closing your eyes, and see what the experience is like for yourself. This will give you a better idea as to how a blind person is likely to want to interact with the UI.

I hope you find this feedback helpful and that your hard work winds up creating a resource that can truly be useful to the target audience and how people will use it.

--Pete

Submitted by peter on Saturday, October 2, 2021

CJ,

One more suggestion I have that could help make this app useful and productive to the blind is to include a complete set of keyboard shortcuts that can be used to perform important functions such as moving from parentheses to parentheses, navigating expressions and sub expressions, etc. Gestures can be kind of limited and accidental touches of the screen can easily change focus to unexpected places. Even sighted users like using keyboard shortcuts to make their work more efficient.

--Pete

Submitted by CJ on Sunday, October 3, 2021

Thank you zeirus-fr and Jenna,

I will add to my list of features:
1. Option to edit in Latex. This will not be the default option. This editing feature may not be available in the first version of the app, but I will do my best.
2. Option to export as Latex.

--CJ

I appreciate your concerns. Thank you for being patient with me. I will absolutely turn on VoiceOver and the screen curtain while I am testing the app. And I will ask blind people to test it and provide feedback.

After reading your feedback, the editing modes feature in my list of features is now:

Provide three options for editing:
Option 1: Edit in 1D linear MathML format. This will be the default option when VoiceOver is on.
Option 2: Edit in 2D format. This might be useful when a blind person is sharing their work with a sighted person.
Option 3: Edit in 1D linear LaTex format. This might not be available in the first version of the app.
It should be possible to switch between editing modes. So, one expression can be worked on in any of the three editing modes.

And I will amend my existing feature relating to a parentheses to parentheses gesture or keyboard shortcut to read:
Include a complete set of keyboard shortcuts that can be used to perform important functions such as moving from parentheses to parentheses, navigating expressions and sub expressions, etc.

I am a great fan of keyboard shortcuts.

--CJ

CJ,

Sounds good. As I said, keyboard shortcuts are very useful even for sighted people!

As a side note, I bought a book many years ago entitled "Windows from the Keyboard". I thought, 'This is great for blind people, I'm glad someone finally put a book together on keyboard shortcuts and how they can be used".

As it turns out, the guy who wrote the book was fully sighted but used to travel a lot on airplanes with those old laptops with the eraser type control to move the mouse. He was so frustrated that he learned to use Windows without a mouse and wanted to share the experience!

Anyway, good luck. Sounds good.

--Pete