Both Facebook and Google have been working hard at using computers and algorithms to identify people in photos. They've gotten really good at it.
We still don't know what they'll do with that technology. To a large degree, it's up to us. But first, we have to understand what's possible.
Why Facebook doesn't need your face
Facebook is one of the leading organizations in the world developing facial-recognition algorithms. Facebook software can now identify people in photographs as well as people can. Facebook's DeepFace (no, I'm not kidding -- it's called DeepFace) can tell whether the subjects in two different photographs are the same person with 97% accuracy. That's even better than the FBI's own Next Generation Identification system.
DeepFace achieves this amazing feat by analyzing faces, turning them into 3D models, then making it possible to recognize the faces from angles and under lighting conditions that are different from those in other photos of the same person. The technology uses more than 120 million parameters, and a page on Facebook's research website explains that the company "trained it on the largest facial dataset to-date, an identity labeled dataset of four million facial images belonging to more than 4,000 identities."
But that's not enough for Facebook. It wants to be able to identify people even when their faces aren't showing. Toward that end, Facebook researchers are developing a system that looks at hairstyle, body shape, posture, clothing and so on.
Facebook can now recognize people whose faces aren't showing with 83% accuracy.
Tellingly, the company tried to avoid freaking people out with this research by developing the algorithm using Flickr pics, not Facebook photos.
While Facebook's ability to recognize people is astonishing, so is Google's.
How Google identifies you without identifying you
Everybody oohed and ahhed at Google I/O last month when Google demonstrated the search feature in its newly announced Google Photos offering.
In fact, we Google+ users have been enjoying this capability for years. Google Photo's search engine can not only tell the difference between cats and dogs, but also identify dog breeds and perform other search feats that seem impossible.
It will even find photos based on adjectives that could be used to describe the images they depict. For example, when I searched my own photo collection using the word delicious, it showed me hundreds of pictures I've taken of foods and beverages that were, in fact, delicious. (It also showed a picture I took during a safari in the Masai Mara in Kenya of a cheetah devouring a gazelle -- I guess that was delicious to the cheetah and Google software somehow knew that.)
Of course, the Google Photos search tool can find people when you search for them. In fact, when you go to the search bar and click to select it, you're immediately presented with three options: People, Places and Things. When you click the More link on the People option, it will show you a picture of every person you have ever photographed -- in order, beginning with the person pictured most frequently.
Click on any of those photos to get all the pictures of that person. When you do that, you'll notice something interesting: Google Photos will show you not only the pictures where the person's face is clearly visible, but also pictures in which the person's face is hardly visible at all.
But unlike Facebook's approach, all the faces that Google Photos search recognizes are visible; I haven't found photos where the person's back is turned.
It's also interesting to note that while Facebook's technology theoretically sounds more advanced, it is still in the research phase and has not been released, whereas Google's search tool is in its shipped product. And it's already available to everyone free of charge.
Google isn't revealing details about how its photo search works, but it probably uses methods that are similar to Facebook's.
One of the most interesting and under-reported features of Google Photos search is that Google has chosen to not associate pictures of people with their identity. For example, when I search for my son Kevin -- who is an active Google user, including a Google+ user -- Google Photos doesn't associate my photos with his identity in its databases.
When I go to Google Photos search, see Kevin's face and click on it, I get hundreds of pictures of him. But when I search for the name Kevin, or for Kevin Elgan, I don't get all the same pictures. Instead, I get pictures that have been tagged or associated with his name directly through mentions on posts.
Obviously, Google could throw an algorithmic switch at any time and start associating people search with identity, but so far it has chosen not to do so.
How Facebook and Google use 'identification by association'
In the old days, facial recognition technology was more straightforward. It would literally analyze faces to look for things like the relative distances between the eyebrows and the nose, and between the bottom of the ears and the chin.
Now, the artificial intelligence behind Facebook's and Google's facial recognition systems is, in fact, recognizing people the way people recognize people. For example, given enough pictures, it actually learns about you. So when it sees your face in one photo, it also makes a note of the setting, the lighting, the clothes you're wearing, your hair and more. When your back is turned and your face is not showing, it can say: "Oh, that's Mike with his back turned."
In the case of Google Photos now, and probably Facebook in the future, facial recognition will also harvest data from social engagement. For example, if I post that my son Kenny is dressed up for Halloween, it can use that information not only to identify him with a mask on, but also to find him in all the other pictures taken of him at the same event with the mask on, but which were not even posted (just automatically uploaded using the Google Photos automated backup feature).
So now what?
There are three important things to note about all this. The first is that research and development on these artificial-intelligence photo recognition technologies will continue and the systems will become far more advanced. It's important for the educated public to grasp the reality of what's possible now, and what will be possible in the future.
In a nutshell, it's only a matter of time before social networks, law enforcement and other organizations will be able to instantly identify any of us with extremely high probability using any photo, including those taken with webcams, security cameras at ATMs and elsewhere, cameras mounted at toll gates, traffic cams and more. Facial-recognition technology is available on more than 28 million mobile devices and that figure is expected to soar to nearly 123 million by 2024.
The second important thing to remember is that the emergence of this technology is not inevitably related to the implementation or abuse of this technology. There seems to be an assumption that it's inevitable that our privacy will be routinely violated in the future. But that's not necessarily true.
The development of technology that can identify everyone all the time is inevitable. But as we've seen with both Facebook and Google, that technology doesn't have to be used to violate our privacy. Facebook is so concerned about the public's reaction that it's not even using Facebook photos to test its latest recognition technology. Google is so concerned about our reaction that it's not associating faces with identities. Clearly, they're both keenly aware of both public concerns and the potential unintended consequences of using this technology to its full potential -- at least for now.
Apple made it clear at its World Wide Developers Conference that it's possible to offer personalization without privacy violation. The company's new Proactive feature for Siri harvests data from email, calendar and more, but the data never leaves the phone and is never associated with a person's ID. It's not uploaded to the cloud or entered into some permanent database. Apple itself never has access to it.
There is no inevitability that personalization or recognition technology must thoroughly violate our privacy. In fact, many of the current privacy violations that happen through our smartphones and computers could be rolled back. The first step in making that happen is for the public to get more sophisticated about the link between what's possible in terms of features and benefits on the one hand and what's necessary in terms of privacy violations on the other.
The third fact worth noting about the ability of companies to recognize you in pictures is that it can be beneficial for your own privacy. For example, it's possible that the search engines of the future could alert you anytime anyone anywhere uses or posts a picture with you in it, even if you're in the background. And you could have the ability to take action when some type of abuse or misuse of such a photo occurs. It's also possible that that type of facial-recognition technology could prevent identity theft, which is becoming increasingly problematic as companies and agencies with our data get hacked.
The day when companies like Facebook and Google will be able to recognize you in pictures with 99% accuracy, even when your face doesn't show, is fast approaching. That capability could lead to a world in which privacy is impossible. But it doesn't have to.
The first step is to understand what's possible and grasp the realm of the technology's implications. The second is for us to ask questions, demand answers and speak up in defense of our privacy.
It's easy for us to throw up our hands in despair and proclaim that our privacy is already dead and gone. But it's not. We can still enjoy the benefits of advanced technology without giving up all our privacy.