[Nfbmt] Facebook automatically adding descriptive alt text to pictures

[beccastewart76 at gmail.com]

As someone who uses Facebook every single day I can tell you that this new technology is fascinating on the one hand and yet disappointing on the other whereas it is not yet able to read text which often if you're appears in the form of a photo. It's very frustrating for me because my friends post hundreds of photos every day that are merely means with words on them and my talking program still cannot read the text. I have high hopes that they will create the technology to do this soon.


Becca Strom- Stewart

> On Apr 5, 2016, at 12:53 PM, Bruce&Joy Breslauer via Nfbmt <nfbmt at nfbnet.org> wrote:
> 
> This may interest some of you.  Joy  
> 
> 
> 
> Ask a member of Facebook’s growth team what feature played the biggest role
> in getting the company to a billion daily users, and they’ll likely tell you
> it was photos. The endless stream of pictures, which users have been able to
> upload since 2005, a year after Facebook’s launch, makes the social network
> irresistible to a global audience. It’s difficult to imagine Facebook without
> photos. Yet for millions of blind and visually impaired people, that’s been
> the reality for over a decade.
> 
> 
> 
> Not anymore. Today Facebook will begin automatically describing the content
> of photos to blind and visually impaired users. Called "automatic alternative
> text," the feature was created by Facebook’s 5-year-old accessibility team.
> Led by Jeff Wieland, a former user researcher in Facebook’s product group,
> the team previously built closed captioning for videos and implemented an
> option to increase the default font size on Facebook for iOS, a feature 10
> percent of Facebook users take advantage of.
> 
> 
> 
> Automatic alt text, which is coming to iOS today and later to Android and the
> web, recognizes objects in photos using machine learning. Machine learning
> helps to build artificial intelligences by using algorithms to make
> predictions. If you show a piece of software enough pictures of a dog, for
> example, in time it will be able to identify a dog in a photograph. Automatic
> alt text identifies things in Facebook photos, then uses the iPhone’s
> VoiceOver feature to read descriptions of the photos out loud to users. While
> still in its early stages, the technology can reliably identify concepts in
> categories including transportation ("car," "boat," "airplane"), nature
> ("snow," "ocean," "sunset"), sports ("basketball court"), and food ("sushi").
> The technology can also describe people ("baby," "smiling," beard"), and
> identify a selfie.
> 
> 
> 
> Last week, I traveled to Facebook’s accessibility lab in Menlo Park to see
> the technology in action. Wieland was there, along with Matt King, a Facebook
> engineer who is blind. King, who was born with limited sight and became blind
> in college, has been advocating for more accessible computers since the
> 1980s. Today, he represents Facebook on a World Wide Web consortium
> responsible for the technical specifications that make web pages accessible.
> 
> 
> 
> The primary way that blind people access the internet is through a screen
> reader — software that describes the elements displayed on a screen (a link,
> a button, some text, and so on) and makes it possible to interact with them.
> The web has evolved over the years to be friendlier to blind people. For
> example, the downward-facing triangle you see on every Facebook post, which
> allows you to hide the post or report it as spam, gets described by the
> screen reader not as a triangle but as as "story options, collapsed pop-up
> button." That way, blind users know they can interact with it.
> 
> 
> 
> But much of the web has long been out of reach for blind people. "You used to
> hear file names, and you didn’t know if they were clickable," King says. "It
> was a big Easter Egg hunt — and it wasn’t any fun at all. Even when I found
> the eggs, a lot of the eggs were photos. People talk in pictures, and talking
> in pictures is inherently out of reach for me." Facebook considered a range
> of approaches to the problem. "We don’t want to add a lot of friction," King
> says. "We could probably require people when they upload a photo: ‘please
> describe this for blind people.’ It would drive people nuts — that would
> never work at scale." (This is the actual approach Twitter is taking to the
> problem, though adding descriptions is optional.)
> 
> 
> 
> Facebook’s scale is enormous: each day, users upload 2 billion photos across
> Facebook, Instagram, Messenger, and WhatsApp. And so the accessibility team
> turned to Facebook’s artificial intelligence division, which is building
> software that recognizes images automatically. "We need a solution to that
> problem if people who cannot see photos and understand what’s in them are
> going to be part of the community and get the same enjoyment and benefit out
> of the platform as the people who can," King says.
> 
> 
> 
> In a demonstration, King pulled up a few stories on Facebook that include
> photos. He set the screen to black so we couldn’t see anything. If you’d like
> to re-think everything you ever thought you knew about web design, watch a
> blind person use the internet for five minutes. King normally has his screen
> reader speak to him incredibly quickly — the slightest audio cues now orient
> him on the page, reading Facebook posts out loud, identifying links, and
> exposing various buttons. His fingers were a blur as he entered commands on a
> standard MacBook Air. I remained totally lost until King turned the screen
> back on, save for the handful of words that described what we were seeing on
> Facebook.
> 
> 
> 
> One Facebook post had a photo with the caption "Sunday night splurge," and
> the description read aloud by the phone was "pizza, food." When King turned
> the screen back on, there was a photo of a giant pepperoni pizza with olives.
> Another photo had the caption "celebrations," and the phone described the
> photo as "three people smiling outdoors." It turned out to be … three people
> smiling outdoors. "Now I’m really understanding the essence of the story,"
> King says. "Sometimes it’s just really amazing what one word can do."
> 
> 
> 
> Facebook is not alone in using machine learning to understand photos; it’s
> one of a few things artificial intelligence can currently do with any level
> of sophistication. Similar technology powers keyword searches in Google
> Photos and Flickr. But the technology is still prone to errors, and millions
> of objects have yet to be parsed. Last year, Google was forced to apologize
> after Photos tagged two black people as "gorillas."
> 
> 
> 
> By default, Facebook will only suggest a tag for a photo if it is 80 percent
> confident that it knows what it’s looking at. But in sensitive cases —
> including ones involving race, the company told me — it will require a much
> higher level of confidence before offering a suggestion. When it isn’t
> confident, Facebook simply won’t suggest a description. "In some cases, no
> data is better than bad data," Wieland says.
> 
> 
> 
> It’s a cliché for tech companies to describe a project as "just the
> beginning," but in this case it feels particularly true. Today it only works
> on one platform, and only in English. There are still millions of objects
> that Facebook can’t recognize with 80 percent confidence. ("Pizza" it knows.
> "Pepperoni pizza with olives" is still a ways away.) But the team is already
> pushing hard on two new tools: recognizing objects in videos, a technology it
> first demonstrated in November; and something it calls "visual Q&A," which
> will allow users to ask questions about pictures and receive an answer from
> Facebook’s AI. You might ask who is in a photo, for example, and it would
> tell you the names of the Facebook friends who appear in it.
> 
> 
> 
> At this stage, automatic alt tags represent a fascinating demonstration of
> technology. But at scale, they could also represent a growth opportunity —
> people with disabilities have been less likely to use Facebook on average,
> for obvious reasons. "Inclusion is really powerful and exclusion is really
> painful," King says. "The impact of doing something like this is really
> telling people who are blind, your ability to participate in the social
> conversation that’s going on around the world is really important to us. It’s
> saying as a person, you matter, and we care about you. We want to include
> everybody — and we’ll do what it takes to include everybody."
> 
> 
> 
> _______________________________________________
> Nfbmt mailing list
> Nfbmt at nfbnet.org
> http://nfbnet.org/mailman/listinfo/nfbmt_nfbnet.org
> To unsubscribe, change your list options or get your account info for Nfbmt:
> http://nfbnet.org/mailman/options/nfbmt_nfbnet.org/beccastewart76%40gmail.com