[BlindMath] Announcing math_scanner, a project for math recognition in images

[Rastislav Kish]

Hello list,

it's most likely not just mine experience, that recognizing math
expressions in image documents has been always difficult, in fact mostly
inpossible.


There are some tools around promising to solve this issue, but none of
them was by any means useful at least for me, as they were unable to
recognize even the most basic formulas.


But then, some time ago, a service called Mathpix has appeared, making
use of deep learning and neural networks to recognize math expressions
and scientific figures. From my tests, its accuracy is truly impressive,
but there is one drawback. You can't feed it an image of the whole page
and get the math, you need to graphically crop only the mathematic
content you want to recognize.


So, a question arises, how could a blind person crop a formula from image?


math_scanner tries to answer this problem. It makes use of the fact,
that OCR engines can recognize the surrounding text around formulas just
fine, with the math content being a clear hole easily detectable by a
blind reader.


math_scanner tracks positions of individual characters and allows the
blind user to use them to surround the math content and get it recognized.


 From early tests, this approach seems to really work, and I was able to
read documents that were totally unreadable by any other tool!


You can read more, as well as get the program on it's repository page:

https://github.com/RastislavKish/math_scanner


I'll be happy to hear your experiences, with learning as well as
results. I found the concept quite difficult to explain, even though
it's natural once you get it, so if you have any questions, feel free to
ask.


Anyway, happy math reading!


Best regards


Rastislav