For organizations hiring hundreds or thousands of people, every interview presents the same challenge: How do you give candidates a consistent opportunity to demonstrate their skills while gathering the evidence needed to make informed hiring decisions?
Structured behavioral interviews offer a research-backed approach to this challenge. They use consistent, job-related questions and clear scoring criteria to give candidates an opportunity to demonstrate their skills. Decades of research in industrial-organizational (I-O) psychology have shown that structured interviews help organizations evaluate candidates and predict job performance (Schmidt & Hunter, 1998; Sackett et al., 2022). Yet delivering structured interviews at scale requires significant time and expertise to develop competencies, create questions, train interviewers, and evaluate responses consistently.
Amazon Connect Talent brings established hiring science to AI-led interviews at scale. Connect Talent uses structured questions, adaptive follow-ups, and evidence-based scoring to help organizations gather consistent, job-related information while giving more candidates the opportunity to demonstrate their strengths.
We built Connect Talent around three core design principles and evaluated how well the service applies them through a combination of internal research and independent expert review. The findings offer a closer look at the science behind Connect Talent’s AI-led, structured interviewing.
Three core design principles
1. Build interviews around job-related evidence
Connect Talent builds each interview around the competencies required for the role. The service uses job requirements and established occupational frameworks to create behaviorally anchored scorecards. Each scorecard defines specific strength and concern signals, and every interview question targets the behaviors it aims to uncover.
The AI interviewer also adapts to each response. After every answer, it checks whether the candidate has provided enough evidence for the competency being assessed. When details are missing, it asks targeted follow-up questions. This approach preserves the structure of the interview while giving candidates more opportunities to explain their experience.
2. Separate interviewing from scoring
Connect Talent separates the interview conversation from the evaluation. The AI interviewer gathers evidence but does not assign scores. Afterward, a separate evaluation process identifies which behaviors the candidate demonstrated and cites the supporting transcript passages.
The service calculates each competency score from 1 to 5 using a fixed formula. When the transcript lacks sufficient evidence, Connect Talent flags the competency instead of producing a score. Hiring teams can review the evidence behind each rating, and missing evidence does not automatically count against the candidate.
3. Build fairness protections into the design
Connect Talent evaluates transcribed candidate responses without using video, voice tone, or physical appearance to calculate competency scores. The service also excludes candidate identity information from scoring and evaluates each competency separately. These safeguards help focus assessments on job-related behaviors and limit the influence of unrelated information.
What the research shows
To understand how well these design principles work in practice, we combined independent I-O psychology evaluations with internal research. We examined whether the service generates job-relevant interview content, identifies behavioral evidence in line with expert assessments, gathers more complete responses through follow-up questions, and distinguishes between levels of demonstrated competency.
Connect Talent keeps interview questions focused on job-relevant skills
The quality of a structured interview starts with the skills it measures. Candidates need questions that reflect the requirements of the role, and hiring teams need confidence that the interview assesses those skills.
Amazon engaged an independent I-O psychology firm to evaluate Connect Talent’s AI-generated interview content. The firm reviewed the service’s baseline output before human editing and examined how well its competencies, questions, and rubrics aligned with job requirements. The evaluation found that Connect Talent retained critical job information in 100% of the job descriptions reviewed. Of the competencies generated, 91% were clearly traceable to the job description. The firm’s psychologists also found that 98% of interview questions targeted the intended competencies, and 94% of rubric signals clearly measured the intended competency.
These findings show that Connect Talent generated job-relevant interview content that met independent expert criteria in a vast majority of cases. Hiring teams can then review, select, and edit the competencies, scorecards, and questions before using them. This human oversight gives organizations control over the interview process while helping them apply structured hiring practices at scale.
Connect Talent’s candidate scoring matched independent expert decisions more than 89% of the time
Relevant questions provide the foundation for a structured interview. The next challenge is evaluating the evidence candidates provide with consistency.
To examine this, an independent I-O psychology firm reviewed 36 interviews assessing whether candidates demonstrated specific strengths and concerns, creating 252 rubric signals.
Connect Talent evaluated the same transcripts, reviewed hundreds of rubric signals, and aligned with expert scoring decisions in 89.3% of cases. Scoring alignment reached 97.6% when candidates provided clear evidence and 92.9% when only considering candidate concern signals, which identify behaviors that may indicate a poor fit for the intended role.
The evaluation also produced a Cohen’s kappa of 0.79, a statistical measure that eliminates random chance coincidence. A Cohen’s kappa of 0.79 is just under the standard of 0.81 for almost perfect agreement. This outcome indicates substantial agreement between Connect Talent and the independent expert candidate scoring.
Follow-up questions give candidates more opportunities to demonstrate their strengths
Candidates may have relevant experience but leave important details out of their first response. In a traditional one-way video interview, a candidate may receive a fixed question and have only one opportunity to answer it. Hiring teams must then evaluate the information available, even when the response leaves many important questions unanswered.
Connect Talent uses adaptive follow-ups to gather more complete evidence. For example, a candidate might describe improving a slow system. The AI interviewer can ask what changed, which actions the candidate took, and how they measured the improvement. These questions help candidates explain their contributions and results in greater detail.
We studied the impact of follow-up questions by comparing assessments based on a candidate’s first response with assessments based on the full interview conversation. In a preliminary evaluation, scores based only on the first response were up to 1.14 points lower on a five-point scale than scores based on the full conversation. Assessments using the full conversation also came closer to expert evaluations.
Without follow-up questions, the scoring process identified fewer strength signals and more concern signals. In some cases, the candidate’s initial response simply lacked the information needed to demonstrate a relevant behavior. The full conversation captured more of what candidates demonstrated and gave hiring teams a more complete record of their experience.
Structured scoring helps hiring teams recognize meaningful differences
Hiring teams need scores that reflect meaningful differences in demonstrated skills. Yet research has found that some AI scoring approaches concentrate ratings within a narrow range, making those differences harder to recognize (Zhu et al., 2026).
Connect Talent uses behaviorally anchored scorecards to evaluate specific strength and concern signals. Rather than asking an AI model to assign an overall rating, the service identifies evidence for each behavior and calculates the competency score using a predefined formula.
Consider the competency “collaboration”. A general AI scoring approach might evaluate a candidate’s overall collaboration skills based on the interview transcript. Connect Talent examines specific behaviors, such as whether the candidate sought input from colleagues before making a decision. Each finding contributes to the final score.
We evaluated this approach across hundreds of expert-rated interview assessments. Connect Talent’s structured scoring used all five points of the rating scale, while holistic LLM-as-judge scoring concentrated its ratings within the middle three points.
This wider distribution allows the scoring system to represent more levels of demonstrated competency. The behaviorally anchored scorecards also help hiring teams understand the evidence behind each rating.
Bringing hiring science to AI-led interviews
Connect Talent brings the rigor of structured interviewing to more people while giving each candidate a meaningful chance to demonstrate their skills. The result is a new opportunity for organizations hiring at scale: more candidates can show what they can do, and more hiring teams can make decisions grounded in the evidence that matters.
Learn More at AWS re:Invent 2026
Join us at AWS re:Invent 2026 for BIZ332: Every Candidate, Every Time: AI-Led Structured Interviews with Amazon Connect Talent, an interactive Chalk Talk led by Sunitha Srivatsa and Gerald Friedland.
The session will explore the science behind Connect Talent, including structured interviewing, adaptive follow-up questions, and evidence-based scoring.
Session 1: Monday, November 30, 2026, 10:00–11:00 AM, Caesars Forum – Level 1, Academy 409.
Repeat session: Thursday, December 3, 2026, 3:00–4:00 PM, MGM Grand, Premier 309.
Register for AWS re:Invent 2026
References
Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068. https://doi.org/10.1037/apl0000994
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research. Psychological Bulletin, 124(2), 262–274.
Booth, B. M., Hickman, L., Subburaj, S. K., Tay, L., Woo, S. E., & D’Mello, S. K. (2021). Bias and Fairness in Multimodal Machine Learning: A Case Study of Automated Video Interviews. Proceedings of the 2021 International Conference on Multimodal Interaction, 268–277. https://doi.org/10.1145/3462244.3479897
Zhu, K., Sheng, Z., Liu, Z., & Tan, J. (2026). Scale Collapse, Model Disagreement, and False Precision: A Multi-Model Audit of LLM Behavioral Interview Scoring.

