Study: Facial Recognition Pros Do Better with AI Partner than with Second Expert

Dr. Alice O’Toole (left) and her team of researchers.

UT Dallas facial recognition researchers, led by Dr. Alice O’Toole, collaborated on a recently published study in Proceedings of the National Academy of Sciences that concluded that forensic experts perform better with an artificial intelligence partner rather than with another person.

O’Toole, Aage and Margareta Møller Professor in the School of Behavioral and Brain Sciences, was senior author of the study, an effort with co-authors from the National Institute of Standards and Technology (NIST), the University of Maryland and the University of New South Wales in Australia.

The experiment sought to determine the most accurate approach to face identification, in terms of participant expertise and methodology. The researchers learned that neither machine nor human gets the best results alone.

“Fusing the most accurate machine with individual forensic facial examiners produced decisions that were more accurate than those arrived at by any pair of human judges or machine judges,” O’Toole said. “The superiority of this human-machine fusion over human-human fusion suggests that humans and machines have different strengths and weaknesses that can be mitigated by cross-fusion.”

Lead author P. Jonathon Phillips, a NIST electronic engineer, described the study as the most comprehensive examination to date of face identification performance across a large, varied group of people and state-of-the-art face recognition algorithms, which have only recently caught up to the performance level of top-performing humans.

“If we had done this study three years ago, the best computer algorithm’s performance would have been comparable to an average untrained student,” Phillips said. “Nowadays, state-of-the-art algorithms are as good as a highly trained professional.”

The study’s 184 human participants included 13 “super recognizers” — those with exceptional natural ability — and 87 trained professional facial examiners. The remaining 84 participants had no such training. Four of the latest computerized facial recognition algorithms also participated; three of those were developed by a University of Maryland team led by electrical and computer engineering professor Rama Chellappa, who contributed to the study.

“We learned that to get the most highly accurate face identification, we should combine the strengths of humans and machines.”

– Alice O’Toole, Aage and Margareta Møller Professor in the School of Behavioral and Brain Sciences

The participants received 20 pairs of face images, which they scored on a seven-point scale in terms of likelihood of each pair being the same person. When the tests were graded, the professionals did significantly better than the untrained people, and the algorithms performed well.

What drew the attention of researchers was that combining the opinions of multiple forensic face examiners — a common strategy in legal cases — did not perform as well as combining one examiner and the best algorithm.

The finding could have significant implications for establishing the strongest methodology for criminal-case evidence.

“Societies rely on the expertise and training of professional forensic facial examiners, because their judgments are thought to be best,” O’Toole said. “However, we learned that to get the most highly accurate face identification, we should combine the strengths of humans and machines.”

Other UT Dallas authors on the paper include postdoctoral researcher Dr. Eilidh Noyes; doctoral students Carina A. Hahn, Ying (Nina) Hu MS’16, Géraldine Jeckeln and Jacqueline Cavazos; and undergraduate student Kelsey Jackson.

The work was funded in part by the Federal Bureau of Investigation, the Office of the Director of National Intelligence and National Institute of Justice.

–Stephen Fontenot