Initially used to improve the experience for visually impaired members of the Facebook community, the company’s Lumos computer vision platform is now powering image content search for all users. This means you can now search for images on Facebook with key words that describe the contents of a photo, rather than being limited by tags and captions.
To accomplish the task, Facebook trained an ever-fashionable deep neural network on tens of millions of photos. Facebook’s fortunate in this respect because its platform is already host to billions of captioned images. The model essentially matches search descriptors to features pulled from photos with some degree of probability.
Facebook isn’t the only one racing to apply recent computer vision advances to existing products. Pinterest’s visual search feature has been continuously improved to let users search images by the objects within them. This makes photos interactive and more importantly it makes them commercializable.
Google on the other hand open sourced its own image captioning model last fall that can both identify objects and classify actions with accuracy over 90 percent. The open source activity around TensorFlow has helped the framework gain prominence and become very popular with machine learning developers.
Facebook is focused on making machine learning easy for teams across the company to integrate into their projects. This means improving the use of the company’s general purpose FBLearner Flow.
“We’re currently running 1.2 million AI experiments per month on FBLearner Flow, which is six times greater than what we were running a year ago,” said Joaquin Quiñonero Candela, Facebook’s director of applied machine learning.
Lumos was built on top of FBLearner Flow. It has already been used for over 200 visual models. Aside from image content search, engineers have used the tool for fighting spam.