Psychological Testing: Chapter 6: Validity

Used in conjunction with the maningfulness of a test score; what the test score truly means; judgment or estimate of how well a test measure what it purports to measure in a particular context

Logical result or deduction

The test has been shown to be valid for a particular use with a particular population of testtakers at a particular time; Validity is within reasonable boundaries of a comtemplated usage

Process of gathering and evaluating evidence about validity; both the test developer and the test user may play a role in the validation of a test for a specific purpose

May yield insights regarding a particular population of testtakers as compared to the norming sample described in a test; necessary when the test user plans to alter in some way the format, instructions, language, or contest of the test;

Content Validity
Criterion-Related Validity
Construct Validity

Construct validity is the umbrella validity;

Content Validity
Criterion-related Validity
Construct Validity

Scrutinize the test’s content
Relate scores obtained on the test to other test scores or other measures
Executing a comprehensive analysis of
How the scores on the test relate to other test scores and measures
How scores on the test can be understood within some theoretical framework for understanding the construct that he test was designed to measure

Relates more to what a test appears to measure to the person being tested than to waht the test actually measures; face validity is a judgment concerning how relevant the test items appear to be

If it appears to measure what it purports to measure what it purports to measure on the face of it

Contributes to a lack of confidence in the perceived effectiveness of the test - with a consequential decrease in the testtaker’s cooperation or motivation to do his or her best

Describes a judgment of how adequately a test samples behavior representative of the universe of behavior that the test was designed to sample

Emerges for the structure of the evaluation; a plan regarding the types of information to be covered by the items, the number of items tapping each area of coverage, the organization of the items in the test, and so forth; represents the culmination of efforts to adequately sample the universe of content areas that conceivably could be sampled in such a test

A method for gauging agreement among raters or judges regarding how essential a particular problem is

Proposed that each rater repond to the following querstion for each item: Is the skill or knowledge measured by this item:
Essential
Useful But not Essential
Not Necessary

Negative CVR - when fewer than half the panelists indicate essential, the CVR is negative
Zero CVR - when exactly half the panelists indicate essential, the CVR is Zero
Positive CVR - when more than half but not all the panelists indicate essential, the CVR ranges between .00 to .99

Judgment of how adequately a test score can be used to infer an individual’s most probably standing on some measure of interest-the measure of interest being the criterion

Concurrent Validity

Predictive Validity

An index of the degree to which a test score is related to some criterion measure obtained at the same time (concurrently)

An index of the degree to which a test score preducts some criterion measure

Relevant
Valid
Uncontaminated

Term applied to a criterion measure that has been based, at least in part, on predictor measures

When test scores obtained at about the same time that the criterion measures are obtained, then the measures of the relationship between test scores and the criterion provide evidence of concurrent validity

Indicated when measures of the relationship between the test scores and a criterion measure obtained at a future time are measured; how accurately scores on the test predict some criterion measure

Validity Coefficient

| Expectancy Data

Correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure; affected by restriction or inflation of range; should be high enough to result in the identification and differentiation of testtakers with respect to target attributes

Used to determine the validity between two measures

Whether the range of scores employed is appropriate to the objective of the corelational analysis
Attrition in number of subjects may occur over the course of the study and the validity coefficient may be adversely affected

The degree to which an additional predictor explains something about the criterion measure that is not explained by predictors already in use

Provide information that can be used in evaluating the criterion-related validity of a test

Shows the percentage of people within specified test score intervals who subsequently were placed in various categories of the criterion; may be created from a scattergram according to the steps listed

Provide an estimate of the extent to which inclusion of a particular test in the selection system will actually improve selection; determining the increase over current procedures

Numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired

Refers to the percentage of people hired under the existing system for a particular position

Draw a scatterplot such that each point in the plot represents a particular test score-criterion score combination; Criterion on Y axis
Draw grid lines in such a way as to summarize the number of people who scored within a particular interval
Counter the number of points in each cell (n)
Count the total number of points within each Vertical interval, this number represents the number of people scoring within a particular test score interval
Convert each cell frequency to a percentage; this represents the percentage of people obtaining a particular test score-criterion score combination; write percentages in the cells; enclose the percentages in parentheses to distinguish them from the frequencies
On a separate sheet, create table headings and subheadings and copy the percentages into the appropriate cell tables
If desired, write the number and percentage of cases per test-score interval; if the number of cases in any one cell is small, it is more likely to fluctuate in subsequent charts; if cell sizes are small, the user could create fewer cells or accumulate data over several years

Entails obtaining the difference between the means of the selected and unselected groups to derive an index of what the test is adding to already established procedures; determines the increase in average score on some criterion measure

Usefulness or practical value of tests

Developed the Decision Theory of Tests

Classification of decision problems
Various selection strategies ranging from single-stage processes to sequential analyses
Quantitative analysis of the relationship bet ween test utility, the selection ratio, the cost of the testing program, and expected value of the outcome
Recommendation that in some instances job requirements be tailored to the applicant's ability instead of the other way around

Tailoring job requirements to the applicant's ability instead of the other way around

Extent to which a particular trait, behavior, characteristic, or attribute exists in the population (expressed as a proportion)

Defined as the proportion of people a test accurately identifies as possessing or exhibiting a particular trait, behavior, characteristic, or attribute

The proportion of people the test fails to identify as having, or not having a particular characteristic or attribute

Amounts to an inaccurate Prediction

False Positive

| False Negative

Miss wherein the test predicted that the testtaker did not possess the particular characteristic or attribute being measured when in fact the testtaker did not

Miss wherein the test predicted the testtaker did not possess the particular characteristic or attribute being emasured when the testtaker actually did

Entails obtaining the difference between the means of the selected and unselected groups to derive an index of what the test is adding to already established procedures; determines the increase in average score on some criterion measure

Usefulness or practical value of tests

Developed the Decision Theory of Tests

Classification of decision problems
Various selection strategies ranging from single-stage processes to sequential analyses
Quantitative analysis of the relationship bet ween test utility, the selection ratio, the cost of the testing program, and expected value of the outcome
Recommendation that in some instances job requirements be tailored to the applicant's ability instead of the other way around

Tailoring job requirements to the applicant's ability instead of the other way around

Employed in ensuring test homogeneity; one item analysis procedure focuses on the relationship between testtakers' scores on individual items and their score on the entire test

Defined as the proportion of people a test accurately identifies as possessing or exhibiting a particular trait, behavior, characteristic, or attribute

The proportion of people the test fails to identify as having, or not having a particular characteristic or attribute

Amounts to an inaccurate Prediction

False Positive

| False Negative

Miss wherein the test predicted that the testtaker did not possess the particular characteristic or attribute being measured when in fact the testtaker did not

Miss wherein the test predicted the testtaker did not possess the particular characteristic or attribute being emasured when the testtaker actually did

Judgment about the appropriateness of inferences drawn from test scores regarding individual standings on a variable called construct

An informed, scientific idea developed or hypothesized to describe or explain behavior; unobservable, presupposed (underlying) traits that a test developer may invoke to describe test behavior or criterion performance

Evidence of Homogeniety
Evidence of changes with age
Evidence of Pretest-Posttest Changes
Evidence of Distint Groups
Convergent Evidence
Discriminant Evidence
Factor Analysis

Refers to how uniform a test is in measuring a single concept

Use of Pearson r to correlate average subtest scores with an average total test score
Reconstruction or Elimination of subtests that in the test developer's judgment do not correlate very well with the test as a whole
For Dichotomously scored test:Eliminating items that do not show significant correlation coefficients with total test scores
For Multipoint Scaled Tests: Items that do not show significant Spearman rank-order corellation coefficients are eliminated
Coefficient Alpha: used in estimating homogeneity of a test composed of multiple choice items

Employed in ensuring test homogeneity; one item analysis procedure focuses on the relationship between testtakers' scores on individual items and their score on the entire test

Tests should reflect progressive changes for constructs that could be expected to change over time

Evidence that test scores change as a result of some experience between a pretest and posttest can be evidence of a construct validity; Any intervening life experience could be predicted to yield changes in score from pretest to posttest

Demonstrating that scores on the test vary in a predictable way as a function of membership in some group; If a test is a valid measure of a particular construct, then test scores from groups of people who would be presumed to differ with respect to that construct should have correspondingly different test scores

Comes from Correlations with tests purporting to measure an identical construct and from correlations with measure purporting to measure related constructs

When a validity coefficient shows little relationship between test scores and/or other variables with which scores on the test being construct-validated should not theoretically be correlated

Experimental Technique that measured both convergent and discriminant validity evidence; matrix or table that results from correlating variables (traits) within and between methods; values for any number of traits as obtained by various methods are inserted into the table, and the resulting matrix of correlations provides insight with respect to both the convergent and the discriminant validity of the methods used

Two or more traits

Two or more methods

Shorthand term for a class of mathematical procedures designed to identify factors or specific variables that are typically attributes, characteristics, or dimensions on which people may differ; employed as a data reduction method in which several sets of scores and the correlations between them are analyzed; identifies the factor or factors in common between test scores on subscales within a particular test or the factors in common between scores on a series of tests

Entails estimating or extracting factors, deciding how many factors to reatin, and rotating factors to an interpretable orientation

A factor structure is explicitly hypothesized and is tested for its fit with the observed covariance structure of the measured variables

Each test is thought of as a vehicle carrying a certain amount of one or more abilities; conveys information about the extent to which the factor determines the test scores or scores

Factor inherent in a test that systematically prevents, accurate, impartial measurement

When a test systematically underpredicts or overpredicts the performance of members of a particular group with respect to a criterion; derived from the point where the regression line intersects the Y-axis

When a test systematically yields significatly different validity coefficient for members of different groups

Numerical or verbal judgment (or both) that places a person or an attribute along a continuum identified by a scale of numerical or word descriptors

Scale of numerical or word descriptors

Judgment resulting from the intentional or uninternional misuse of a rating scale

Error in rating that arises from the tendency on the part of the rater to be lenient in scoring, marking, and/or grading

Opposite of Leniency/Generosity Error; when tests are scored very critically by the scorer

The rater exhibits a general and systematic reluctance to giving ratings at either the positive or negative extreme; all the rater's ratings would tend to clusted in the middle of the rating continuum

(Central Tendency, Leniency, Severity Errors) overcome through the use of Rankings

A procedure that requires the rater to measure individuals against one another instead of against an absolute scale; By using rankings, the rater is forced to select first, second, third choices, etc.

Describes the fact that, for some raters, some ratees can do no wrong; a tendency to give a particular ratee a higher rating than he or she objectively deserves because of the rater's failure to discriminate among conceptually distinct and potentially independent aspects of a ratee's behavior

The extent to which a test is used in an impartial, just, and equitable way