We have changed and adapted how we manage test integrity a lot over the past 20 years.
Has your assessment done the same?
In my first ten years doing this work, we caught one cheater. Today, 83% of candidates say they would use AI on an assessment if they thought no one could tell. Most vendors are still using 2006 methods to catch them.
In 2006, a candidate for a director-level role sat at a shared desktop, worked through a paper-style personality test in a browser window, and finished in ninety minutes. No one checked who typed the answers. It was rare for someone to find another person to answer their assessment for them.
Twenty years later, a similar candidate takes the same test and finishes in under an hour. The colors and the design may be different, but it is the same point-and-click interface. In 2006 there was no AI agent to help you with these assessments.
When sharing this process with one of our clients, he said, “We’ve had a similar dynamic with our junior banker test-taking. It’s amazing how people leverage these tools to cheat.”
It’s happening everywhere, and it is important to know when cheating is happening.
14%
Of candidates admit to using AI on their assessments
83%
Would if they thought no one could tell
46%
Of resumes contain at least one discrepancy from a background check
No one should be surprised that the current process does not give an accurate view of a candidate.
Candidates have become so adept at hacking the system that organizations are getting the results they manufactured.
No one should be surprised that the data is manipulated either. In one Stanford study, 90 percent of job seekers admitted lying at least once during an interview. Platforms that still rely on self-report are about as accurate as a coin flip. The assessment industry has become naive enough to believe its impression-management scales are sufficient to tell whether someone is gaming the system.
To make sure the money you spend on assessments is buying accurate results, ask your assessment companies how they manage the manipulators. They should be using at least two of the three stop gaps below.
Watching the camera is only half of it. The other half is the mouse movement.
An assessment company in South Africa was one of the first to take proctoring seriously. They hired a consultant to sit on the other end of a Zoom screen and watch the candidate take the assessment. Effective, perhaps, scalable, definitely not.
Building a proctoring system that is both effective and scalable takes time. Unless it matters a great deal to the assessment company, they are unlikely to make the investment. At Psybil we have invested in two processes.
HOW PSYBIL® PROCTORS AN ASSESSMENT
- An AI that watches the camera and notes when a candidate shows unusual behavior.
- Mouse-movement tracking because it is just as easy to open a second tab and ask A it is to have someone else in the room.
When our algorithms flag unusually high cognitive scores, or a candidate who appears uniquely virtuous, we go back to the record and establish whether they responded truthfully.
Really good liars believe their own lie.
Twenty years ago, assessments tried to catch cheaters by asking the same question over and over, assuming a cheater could not remember their previous answers and would get caught in an inconsistency.
But the best liars tend to be highly consistent even when inaccurate. This lazy approach only made assessments longer and frustrated the people answering honestly.
The solution lies in probability theory. All of us have good traits and not-so-good traits, and it is unlikely that any one person holds positive traits across multiple domains. If a test taker endorses an item that is true of only 5% of the population, you do not need to be good at math to see that holding many such traits is highly improbable.
You cannot game a test when you cannot tell what it is measuring.
When Dr. Popple first started building Psybil in 2006, the only way he knew to select items was principal component analysis. The statistic required ten times or more the amount of data, then clustered similar items together. Some items clearly connect to a subscale; others are obvious only to someone trained in psychology or cognitive theory.
One of the first people to work in business development for Psynet Group came to us from a client. He would tell prospective clients that when he first took the assessment, he tried hard to show he was detail- oriented, even though he wasn’t. When the results came back, his lack of detail orientation was clear.
Psychometricians call this face validity. When an assessment has high face validity, it is easy to see what it is trying to measure. When it has low face validity, as with Psybil, it is much harder to work out what is being captured, and much harder to game.
Relying on 2006 methods isn’t enough to catch cheaters using 2026 technology.
AI has given cheaters the tool they need to manipulate assessment results. The question is no longer whether candidates are gaming your assessment; it is whether your vendor can tell.
Ask your vendors one question: how are you using AI to fight AI in assessment fraud? Then check whether they use at least two of the three stop gaps above.