Snoopers beware: Google's AI can now spot shoulder-surfers peeking at your screen

Google researchers have developed a privacy application that can instantly detect when a stranger glances at your screen over your shoulder.

Researchers Hee Jung Ryu and Florian Schroff will demonstrate their shoulder-surfing warning system at next month's Neural Information Processing Systems Conference in California.

The pair haven't released a paper that explains technology, but a video demonstration suggests they've installed a lightweight machine-learning model on a Pixel smartphone that uses its front-facing camera for rapid gaze detection.

When the user holds a phone up to chat or view a private video, say, on a train or other crowded place, the algorithm will detect when someone else from behind begins looking at the screen too. The video shows that the algorithm can detect a stranger's gaze within milliseconds.

The video suggests the researchers are using FaceNet, a facial-recognition neural network developed by Schroff and other Google researchers in 2015, as well as GazeNet, a gaze-estimation neural network recently described by researchers in Japan and Germany.

The two Google researchers call their application an 'electronic screen protector', which combines the two areas of computer vision research to provide enhanced privacy when using a big-screen smartphone in public spaces.

The application builds on face-authentication systems already on smartphones by using the enrolled face to distinguish between the user and a stranger, while the gaze-estimation model helps detect the stranger's gaze almost instantaneously.

The system takes just two milliseconds to detect a stranger's gaze, and 47 milliseconds to recognize a face. It also takes 115 milliseconds to detect a face in each frame.

"Because of the quick, robust, and accurate gaze detection mobile model, we can now easily identify the face identity and gaze simultaneously in real time," they explain in a brief description.

"Hence, the application, an electronic screen protector, can enable its enrolled users to continue reading private and confidential contents on your mobile device, while protecting their privacy from onlookers in a crowded space such as the subway or an elevator."

The system takes two milliseconds to detect a stranger's gaze, and 47 milliseconds to recognize a face.

Google/YouTube

Previous and related coverage

iPhone X has face recognition but this heart-scan authentication goes one better

A heart-based authentication system keeps you logged in until you walk away from the device.

Face ID is so good you won't miss the Home button

The removal of the iPhone's Home button has required some tradeoffs, but the superior usability of Face ID sets a new standard for secure access.

iPhone X: Sorry Apple, but I just can't face using Face ID

The iPhone X is putting facial recognition front and centre -- so why does it leave me frowning?

Read more about facial recognition

• Facial recognition technology to replace passports at Australian airports
• A Stormtrooper robot with facial recognition technology
• Georgia rolls out NEC's facial recognition technology across major cities
• How to set up Windows Hello facial recognition on your laptop [CNET]
• Want to unlock the iPhone X? Just stare at it [CNET]