PSY 3630 Module 2 Selection Methods Paper: Weighing Validity and Adverse Impact Example

Which Tools Predict and Which Exclude: Choosing How to Hire Member Service Representatives

Student Name

B.S.B.A. in Human Resources/Organizational Management, Thomas Edison State University

PSY-3630: Industrial Psychology

Mentor Name

Month Day, Year

Which Tools Predict and Which Exclude: Choosing How to Hire Member Service Representatives

Introduction

The job analysis in module one identified ten characteristics member service representatives need on their first day, led by emotional control with upset members, active listening, accuracy with numbers and reading complex policy. The credit union currently hires using an application form and one unstructured interview with a branch manager. With 38 percent turnover and uneven performance, the human resources director asked which methods would predict performance best without screening out qualified applicants unfairly. This paper compares the evidence and recommends a hiring process.

What Validity Means

A selection method's criterion-related validity is the correlation between applicants' scores and their later job performance. A correlation of zero means the method predicts nothing; higher values mean better prediction. Because research samples are often restricted, for example by studying only people who were hired, meta-analyses correct observed correlations for range restriction and for unreliable performance ratings. How much to correct has been debated.

The 1998 Estimates

Schmidt and Hunter (1998) summarized meta-analyses covering 85 years of research. Their estimates placed general mental ability tests at a validity of .51 for job performance, work sample tests at .54, structured interviews at .51, unstructured interviews at .38, integrity tests at .41 and conscientiousness measures at .31. The authors then asked what each method contributed on top of a cognitive test, finding the biggest gains from adding an integrity test or a structured interview. For decades these numbers made cognitive ability tests the default recommendation.

The 2022 Revision

Sackett et al. (2022) argued that past meta-analyses applied range restriction corrections too heavily, applying large corrections to studies where restriction was small or absent. Re-estimating with more conservative corrections, they found lower validities for many methods. Structured interviews emerged as the most valid at about .42, followed by job knowledge tests and empirically keyed biodata. General mental ability tests fell to about .31, and conscientiousness to about .19. Unstructured interviews remained well below structured ones. The authors concluded that the gap between cognitive tests and other methods had been overstated.

The Diversity-Validity Dilemma

Validity is not the only concern. Of common predictors, cognitive ability tests produce the biggest gaps in average scores across racial and ethnic groups, roughly one standard deviation between Black and white applicants, which can produce adverse impact, a substantially lower selection rate for one group. Ployhart and Holtz (2008) reviewed sixteen strategies for reducing these differences while keeping validity. The most effective included measuring the full range of job-relevant characteristics rather than cognitive ability alone, adding noncognitive predictors such as structured interviews and personality measures, which show smaller group differences, and reducing reading demands that are unrelated to the job. Strategies such as score banding or changing test instructions had smaller effects.

Fitting Methods to the Job

The job analysis matters here. Emotional control and listening, the most important first-day characteristics, are measured better by a structured interview and a work sample than by a cognitive test. Accuracy with numbers and reading policy can be measured with a short, job-specific test of basic math and reading rather than a general ability test, which keeps the content related to the work and lowers unrelated reading demands.

Recommended Process

The credit union should use a four-step process. First, an online application with a few empirically checked biodata items, such as prior customer-facing experience. Second, a 20-minute job-specific test of numerical accuracy and policy reading, using passages from the credit union's own procedures. Third, a short work sample in which applicants handle two recorded member calls, one routine and one with an upset member. Fourth, a structured interview built from the job analysis, described in module three. Scores from the last three steps would be combined using set weights rather than a manager's overall impression.

Monitoring Fairness and Results

The human resources team should calculate selection rates by group each quarter and compare them with the four-fifths guideline used by federal agencies, looking into any step at which a group is where one group's pass rate falls short of four-fifths of the best-performing group's rate. It should also track new hires' performance ratings and turnover at six and twelve months to confirm the methods predict as expected.

Cost and Applicant Reactions

Methods also differ in cost and in how applicants react. The job-specific test and recorded calls can be delivered online for a few dollars per applicant, while each structured interview takes an hour of two staff members' time, so it comes last, after cheaper steps have narrowed the pool. Applicants generally view work samples and interviews as fair because they look like the job, while abstract tests can seem unrelated. Positive reactions matter in a tight labor market, since good applicants who feel misjudged may withdraw.

Why Not Interview Alone

Some managers asked why the credit union does not simply improve its interview and skip the other steps. The answer is that each method captures something different. The test measures accuracy and reading under time pressure, the recorded calls show how an applicant actually responds to an upset member and the interview probes past behavior. Combining methods that correlate only modestly with each other adds prediction beyond any one alone.

Next Steps

The human resources team will pilot the new process in four branches for six months before rolling it out, comparing new hires' early performance and turnover with those hired the old way in the other branches.

Limits

Meta-analytic validities are averages across many jobs and settings, and local results may differ. The 2022 estimates themselves may be debated as further research appears. The credit union and its process are invented.

Conclusion

Schmidt and Hunter's estimates long favored cognitive tests, but Sackett and colleagues' correction placed structured interviews first. Ployhart and Holtz showed that measuring the full range of job requirements with noncognitive methods reduces adverse impact. For member service representatives, a job-specific skills test, two recorded member calls and a structured interview, combined by formula and monitored for fairness, fit both the evidence and the job.

References

Ployhart, R. E., & Holtz, B. C. (2008). The diversity-validity dilemma: Strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection. Personnel Psychology, 61(1), 153-172. https://doi.org/10.1111/j.1744-6570.2008.00109.x

Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040-2068. https://doi.org/10.1037/apl0000994

Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262-274. https://doi.org/10.1037/0033-2909.124.2.262