Don't Start with AI, Start with the Problem
By Saurabh Rana and Clodagh O'Reilly.
An interesting discussion around a recent PMaps assessment case study prompted a broader question: in our enthusiasm to adopt AI, are we sometimes starting in the wrong place?
AI is difficult to avoid in conversations about the future of work.
In recruitment alone, candidates are using generative AI to write CVs, applications and responses to assessment questions, while employers are increasingly using technology to screen applications, rank candidates and support selection decisions. Gartner reported in 2025 that 39% of candidates had used AI during the application process. Among those using it, 54% had generated résumé content, 50% cover-letter content and 29% answers to assessment questions.
Understandably, organisations are asking what they should be doing with AI.
But perhaps that is the wrong place to start.
Business Psychology has a well-established alternative:
Start with the problem. Diagnose before intervening.
The principle is hardly new. Effective diagnosis requires us to understand what is happening, establish what we are trying to change and identify the evidence that would demonstrate improvement before selecting an intervention.
Why should our approach to AI be any different?
When an AI Discussion Becomes a Diagnostic One
A recent case study shared with us by assessment provider PMaps illustrates the point particularly well.
Our initial discussion concerned how technology could support the development and refinement of psychometric assessments. That naturally led to questions about AI: Where was it being used? What was it doing? What remained the responsibility of the psychologist?
But exploring those questions revealed something potentially more interesting.
The techniques being used by PMaps in the case were Exploratory Graph Analysis (EGA), Unique Variable Analysis (UVA) and bootstrap EGA. These are computational psychometric techniques rather than a separate generative-AI tool, LLM or AI system.
That distinction matters.
It is increasingly easy to use AI as shorthand for sophisticated technology, automation or algorithmic analysis. But not every computational technique is AI; and something does not have to be AI to represent an advance in practice.
The better question was therefore not “How is PMaps using AI?”
It was “What problem was PMaps trying to solve?”
Illustrated In Practice: Reducing Unnecessary Assessment Time
One problem was straightforward: assessments need to collect sufficient evidence to support sound decisions, without asking candidates to spend unnecessary time completing them.
In one internal PMaps study, a 22-item scale was analysed using data from 588 participants. Traditional EFA/CFA analysis suggested three possible dimensions, although the third was borderline and two factors correlated strongly (r = .887), raising questions about whether they represented genuinely distinct constructs.
EGA supported a two-dimensional structure, while UVA identified a potentially redundant item pair. One item was subsequently removed.
The resulting 21-item instrument retained strong reliability across its two dimensions (ω = .905 and .841), with broadly comparable model fit despite fewer parameters.
The reduction from 22 items to 21 is modest. It would therefore be inappropriate to present this case as evidence of a dramatic technological transformation, or to claim that the resulting assessment became more predictive.
But that isn't necessarily the point.
PMaps reports that reducing overlapping items across its assessment work has enabled shorter candidate completion times. The practical objective is therefore identifiable: remove unnecessary assessment burden while retaining psychometric defensibility.
Technology supported the solution. It wasn't the objective.
Better Has To Mean Something
This distinction becomes important when organisations talk about “using AI to improve” recruitment or assessment.
What does improve mean?
It might mean reducing candidate completion time. It might mean identifying redundant questions, reducing development costs or enabling greater scale. Elsewhere, the objective might genuinely be stronger reliability, validity or predictive accuracy.
Those outcomes are not interchangeable.
A 30-minute assessment reduced to 20 minutes while retaining the properties necessary for its intended use may represent a meaningful improvement in efficiency and candidate burden. But it should not be described as more accurate unless evidence demonstrates greater accuracy.
Diagnosis gives us the discipline to define the desired outcome first, and evaluate the technology against it.
Illustrated In Practice: Technology Doesn't Remove Judgement
A second PMaps case study reinforces this point.
An initial pool of 60 items was refined into a 30-item sales assessment and tested with 817 financial-services candidates. Two dimensions emerged: Process Orientation and Result Orientation.
The evidence was mixed. Process Orientation demonstrated strong composite reliability (.91), while Result Orientation was weaker (.66). Average variance extracted was marginal for both dimensions, and model-fit statistics were mixed.
That is useful evidence precisely because it isn't perfect.
A sophisticated computational method can identify patterns, investigate dimensionality and support more efficient development. It cannot make the resulting instrument psychometrically sound simply by virtue of having been used.
PMaps also confirmed that the technology in this process does not make candidate pass/fail decisions; assessments remain norm-based. Their earlier description also places construct judgement with the psychometric team.
Technology can augment professional practice. It does not remove the responsibility to exercise professional judgement.
AI Literacy Doesn't Require Becoming an AI Specialist
There is a broader lesson here for organisations considering AI.
You do not necessarily need to become an “AI shop” to benefit from technological advances. Nor should adopting AI become evidence, in itself, that an organisation is innovating.
Instead, practitioners need sufficient technological literacy to distinguish between AI, automation, computational analysis and established statistical methodologies, and sufficient professional expertise to know when each is useful.
That suggests a different sequence for adopting technology. Have you considered:
- What is the problem?
- What outcome are we trying to improve?
- What evidence will tell us whether we have succeeded?
- What technology or methodology is best suited to helping us achieve it?
- What remains a matter for human judgement?
Only then does the question become: could AI help?
For Business Psychologists, there is something reassuringly familiar about that approach.
The technologies may be changing rapidly. The fundamental discipline does not have to. Hence this perspective: Don't start with AI. Start with the problem.
For ABP Members:
Not a member? We invite you to join us at The Association for Business Psychology!
About the Authors
Saurabh Rana is the founder of PMaps, and has been running the company for close to a decade. He is a double post graduate in Finance and Marketing, and also completed his advance degree in Industrial Psychology from TISS. He oversees psychometric practices at PMaps and is involved in taking client requirements, calibrating personality inventory, periodically testing scientific properties, and running both face and criterion validity.
Clodagh is a Certified Principal Business Psychologist and assessment specialist, who worked as an in-house subject matter expert, delivering innovative and highly effective selection solutions for global organisations, before leading the IBM Smarter Workforce assessment consultancy in EMEA, and then taking on a role in the US supporting client deployment of IBM's artificial intelligence solutions for HR. She has also published six case study collections on behalf of The ABP, showcasing the practical value of business psychology at work.
References:
Christensen, A. P., & Golino, H. (2021). Estimating the stability of psychological dimensions via bootstrap exploratory graph analysis: A Monte Carlo simulation and tutorial. Psych, 3(3), 479–500. https://doi.org/10.3390/psych3030032
Gartner, Inc. (2025, July 31). Gartner survey shows just 26% of job applicants trust AI will fairly evaluate them. https://www.gartner.com/en/newsroom/press-releases/2025-07-31-gartner-survey-shows-just-26-percent-of-job-applicants-trust-ai-will-fairly-evaluate-them
Golino, H. F., & Epskamp, S. (2017). Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research. PLOS ONE, 12(6), Article e0174035. https://doi.org/10.1371/journal.pone.0174035
PMaps. (2026). Validating a bifactorial behavioral model for pre-employment sales assessment [White paper]. PMaps. https://www.pmapstest.com/case-study/behavioral-model-sales-research
ResumeBuilder.com. (2024, October). 7 in 10 companies will use AI in the hiring process in 2025, despite most saying it's biased. https://www.resumebuilder.com/7-in-10-companies-will-use-ai-in-the-hiring-process-in-2025-despite-most-saying-its-biased/
TopResume. (2025, June 3). Survey: Where employers draw the line on the use of AI in hiring [Blind test survey of 600 U.S. hiring managers]. https://topresume.com/career-advice/ai-in-hiring-survey