Facebook pumps up character recognition to mine memes
Facebook described how they applied convolutional neural networks, already widely used for image recognition, to detect whether there's text in an image and then transcribe that text. It could be useful for helping visually impaired users, though the immediate appeal of the system would seem to be to mine the vast quantities of internet memes being uploaded by the millions.
By Tiernan Ray | September 11, 2018 -- 21:30 GMT (14:30 PDT) | Topic: Artificial Intelligence
No more interference: Facebook is a building a war room ahead of US midterms Facebook is planning to establish a physical "war room" designed to bring staff together to find and destroy attempts to meddle with upcoming elections.
More Prime Day
How and when to find the best deals
Business Bargain Hunter's top picks
Amazon's business strategy behind the sale
Best deals: Smartphones
Best deals: Windows 10 laptops
Best deals: Storage, SSD, and flash drives
Researchers at Facebook offered up a summary of a system they call "Rosetta," a machine learning approach that boosts traditional optical character recognition, or "OCR," to mine the hundreds of millions of photos uploaded to Facebook daily.
Say you want to search for memes in images on Facebook: The site's challenge is to detect whether there are letters printed within an image, and then parse those letters to know what a phrase says.
Also: Facebook accidentally exposes 14 million users' private posts
This technology has, of course, been in use for document processing for ages, but the challenge at Facebook was both to recognize text in any number of complex images, including text laid over the image, as in an internet meme or text such as a sign that was part of the original image, and then to make it work at the scale of the site's constant stream of images.
Facebook researchers Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar shared the work on Rosetta at the Knowledge Discovery and Data Mining conference in London in late August, in a formal paper, and today, two of the authors, Gordo and Sivakumar, along with Facebook's Manohar Paluri, offered up a somewhat simpler blog post describing the work.
Facebook split up the task of "extracting" text from an image into two separate matters, that of first detecting whether there is text at all in an image, and then to parsing what that word of phrase might be.
Also: If Facebook worked we wouldn't be in this mess
For the first task, detection, the authors used a convolutional neural network (CNN) called "Faster R-CNN," which itself derived from work done originally by Facebook's Ross Girshick when he was at Microsoft. While CNNs have been used quite a bit in the last decade for image recognition tasks, such as ImageNet, the R-CNN adds the notion of "regions" as a way to speedily pick out objects in an image and say where precisely in the image the object is located.
Facebook has already widely deployed an object-recognition system throughout its infrastructure called "Detectron," and having that in place clearly helped in this case.
Once text is located in an image, the coordinates of that image are passed to another CNN to discern the word or phrase, character by character. The product of that second step are sequences of characters making up words and phrases.
Also: Facebook's new Rosetta AI system helps detect hate speech CNET
Because recognizing long words or long phrases can be especially challenging, the authors describe using what's called a "curriculum" approach to train the character recognition system. They started out by training the system on small words of five characters or less, and progressively increased the length of words with subsequent iterations of the training.
All the training work for both the detection part and the recognition part were performed using the "Caffe2" framework.
The authors spend a substantial amount of time in the original paper describing how they tuned the system for optimal speed for "inference," when a new photo is looked at and has to be quickly searched for text and transcribed. "Given our scale and throughput requirements, we spent [a] significant amount of time improving the execution speed of text detection model while keeping the detection accuracy high," they write.
Also: How to protect yourself on Facebook TechRepublic
The Rosetta system is currently operational in Facebook's network, being used daily, the authors write. The authors offer that future challenges will include pulling text out of video applications.
How to reduce the amount of information you... SEE FULL GALLERY
Previous and related coverage:
Should Mark Zuckerberg be fired? Security professionals have their say
Does the Cambridge Analytics scandal -- and all the other Facebook privacy snafus -- warrant the removal of its CEO? A survey of security professionals offers their view.
Facebook reveals new covert efforts to sway 2018 midterm elections
The social network isn't saying who's behind the campaign, noting that it doesn't have the technical evidence at this time to confidently point a finger.
Facebook's 'war room' hunts and destroys election meddling, fake news
The physical room will be tasked with protecting the network against insidious attempts to tamper with the US midterm elections.
Social Enterprise Digital Transformation CXO Internet of Things Innovation Enterprise Software
More from Tiernan Ray
Chip world tries to come to grips with promise and peril of AI
Machine learning goes beyond theory to beat human poker champs
A.I.'s Minority Report for retail: They know you'll return it even before you buy it
IBM launches open ML dataset hub curated for the enterprise
The IBM Data Asset eXchange provides free and open datasets for machine learning, with quality and licensing term checks that should make it simpler for businesses to use. ...
Enterprise organizations plan to double AI deployments by 2020
Over half of organizations already have at least one AI or ML project in place.
Chip executives gathered in San Francisco to discuss what to do about the explosion in demand for deep learning forms of A.I. that is pushing at the limits of today's chips. Possibilities ...
On Delta, facial recognition boarding doesn't sound optional
I listened as the gate agent tried to excite passengers about facial recognition boarding. He didn't mention they could opt out. This is how technology forces itself upon you. ...
Advice for technologists on promoting AI ethics
AI ethics is good for business. Time for IT managers and professionals to step up and take a leadership role.
Now Microsoft wants to use AI to save history
Microsoft creates a program that aims to use AI to preserve the world's cultures in the face of globalisation.
How to deal with a breakdown in theoretical support in machine learning? Researchers from Carnegie Mellon and Facebook describe winning many hands against the world's top poker ...