Psychological Testing: Chapter 7: Utility

Usefulness or practical value of testing to improve efficiency; used to refer to the usefulness or practical value of a training program or intervention

Reliability and valibidy of a test; reliability and validity coefficients are acceptably high

Financial cost of the selection device under study; expenditures associated with testing or not testing

A particular test
A supply of blank test protocols
Computerized test processing, scoring, and interpretation from the test publisher or some independent service

Payment to professional personnel and staff associated with test administration, scoring, and interpretation
Facility rental, mortgage, and/or other charges related to the usage of the test facility
Insurance, legal, accounting, licensing, and other routine costs of doing business

Profits, gains, or advantages accrued from testing

Increase in quality of workers’ performance
Increase in quantity of workers’ performance
Decreases in time to train workers
Reduction in number of accidents
Reduction in worker turnover

Broadly defined as a family of techniques that entail a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment

Utility analysis is an umbrella term covering various possible methods, each requiring various kinds of data to be inputted and yielding arious kinds of output

One test is preferable to another test for use for a specific purpose
One tool of assessment (such as a test) is preferable to another tool of assessment (such as behavioral observation) for a specific purpose
The addition of one or more tests (or other tools of assessment) to one or more tests (or other tools of assessment) that are already in use is preferable for a specific purpose
No testing or assessment is preferable to any testing or assessment

One training program is preferable to another training program
One method of intervention is preferable to another method of intervention
The addition or subtraction of elements to an existing training program improves the overall training program by making it more effective and efficient
The addition or subtraction of elements to an existing method of intervention improves the overall intervention by making it more effective and efficient
No training program is preferable to a given training program
No intervention is preferable to a given intervention

Expectancy Data

Brogden-Cronbach-Gleser Formula

Will show that a test is working as it should and, by instituting this new test on a permanent basis, the company could reasonably expect to improve its productivity

Provides an indication of measure - an interval that may be categorized as parring, acceptable, or failing

Developed a formula for the dollar amount of a utility gain resulting from the use of a particular selection instrument under specified conditions

Refers to an estimate of the benefit (monetary or otherwise) of using a particular test or selection method, taking into consideration the following factors:
number of applicants selected per year
Average length of time in the position
Criterion-Related Validity coefficient for the given predictor and criterion
Standard deviation of performance in Dollars or employees
Mean (standardized) score on the test for selected applicants

Refers to an estimated increase in work output

Pool of Job Applicants
Complexity of the Job
Cut Score in Use

Issue of how many people would actually accept the employment position offered to them even if they were found to be a qualified candidate; Estimates of people who will actually accept the position should be adjusted downward as much as 80% in order to provide a more realistic estimate of the utility of a tool of assessment used for selection purposes

The more complex the job, the more people differ on how well or poorly they do that job

Numerical reference point derived as a result of a judgment and used to dived a set of data into two or more classifications, with some action to be taken or some inference to be made on the basis of these classifications

Reference point, in a distribution of test scores used to divide a set of data into two or more classifications - that is set based on norm-related considerations rather than on the relationship of test scores to a criterion

Set with reference to the performance of a group (or some target segment of a group)

Reference point - in a distribution of test scores used to divide a set of data into two or more classifications - that is typically set with reference to a judgment concerning a minimum level of profieciency required to be included in a particular classification;

Refers to the use for the purpose of categorizing testtakers

At every stage, a cut score is in place for each predictor used; Cut score used for each predictor will be assigned to ensure that each applicant possess some minimum level of a specific attribute or skill; assumes that an individual must possess a certain minimum amount of knowledge, skill, or ability for each attribute measured by a predictor to be successful in a desired position

One collective element of a multistage decision-making process in which the achievement of a particular cut score on one test is necessary in order to advance to the next stage of evaluation in the selection process

Completeness of the filling out of the form may be used to disqualify the applicant

Individuals with low test scores, GPA's or poor letters of recommendation are eliminated

Entails unique demands to be successfully met

An assumption is made that high scores on one attribute can balance out or compensate for low scores on another attribute; a person strong in some areas relevant to the position in question

Devised the Angoff Method

Method for setting fixed cut scores applicable to personnel selection tasks as well as to questions regarding the presence or absence of a particular trait, attribute, or ability; judgments of experts are averaged to yield cut scores for the test; weak when there is low inter-rated reliability and major disagreement regarding how certain populations of testtakers should respond to items

Entails the collection of data on the predictor of interest from groups know to possess, and not to possess, a trait, attribute, or ability of interest

Cut scores are typically set based on testtakers' performance across all items on the test; some portion of the total number of items on the test must be scorred correct in order for the testtaker to pass the test; each item is associated with a particular level of difficulty; in order to pass the test, the testtaker must answer items that are deemed to be above some minimum level of difficulty, which is determined by experts and serves as the cut score

Entails the arrangement of items in a histogram, which each column in the histogram containing items deemed to be of equivalent value; the process involves several rounds of judgments in which experts may receive feedback regarding how their ratings compare to ratings made by other experts

Typically used in academic applications; begins with the training of experts with regard to the minimal, knowledge, skills, and/or abilities that testtakers should possess in order to pass; Level of difficulty to use as the cut score is decided upon by the test developers

Experts are trained with regard to the minimal knowledge, skills, and/or abilities that testtakers should possess in order to pass
Experts are given a book of items with one item printed per page, items are arranged in an ascending order of difficulty
Experts place a bookmark between two pages that are deemed to separate testtakers who have acquired the minimal knowledge, skills, and/or abilities from those have not
Bookmark serves as the cut score
Additional rounds of bookmarking with the same or other judges may take place as necessary
Feedback regarding placement may be provided, and discussion among experts about the bookmarking may be allowed
The level of difficulty to use as the cut score is decided upon by the test developers

Training of experts
Possible floor
Ceiling effects
Optimal length of item booklets

Proposed a norm-referenced method for setting cut scores; called the Method of predictive yield

Proposed the use of regression for setting cut scores when criterion-related data are available

Family of statistical techniques typically used to shed light on the relationship between certain variables (such as scores on a battery of test) and two (and in some cases more) naturally occuring groups (such as persons judged to be successful on the job and persons judged not to be successful on the job