Psychological - Lecture 2 Validity & Reliability (Catherine) Part 3

Predictive Validity is an index of the degree to which a test score predicts some criterion measure (in the future)

By obtaining test scores now, and criterion measure in the future using multiple predictors.

-considering the validity coefficient in the context of corresponding issues including

-incremental validity and expectancy data

NEEDS work - not clear

The Validity coefficient is a correlation that provides a measure of the relationship between test scores and scores on the criterion measure.

Uses more than one predictor

Additional predictors used in ascertaining criterion-related predictive validity should possess incremental validity.

That is the degree to which an additional predictor explains something about the criterion measure that is not explained already by predictors already in use.

Expectancy Data provides useful information to evaluate the criterion-related validity of a test.

Using a score obtained from one test or measure expectancy tables illustrate the likelihood that the test-taker will score within some interval of scores (such as pass or fail).

An expectancy table can be created by a scatterplot according to

An expectancy table shows the relationship between scores on e.g. high school exams and relationship to university grade

Construct Validity is concerned with how well inferences drawn from a test score relate to current theories or knowledge (i.e. constructs)

Evidence of Homogeneity

Evidence of Changes with Age

Evidence of Distinct Groups

Convergent Evidence****

Discriminant Evidence****

Factor Analysis

How uniform the test is for measuring a single concept

Correlate sub-sections with the whole test score

Item analysis

How important is homogeneity?

Construct Validity executes a comprehensive analysis of how scores on the test:

a. relate to other scores and measures

b. can be understood within some theoretical framework for understanding the construct that the test was designed to measure

-Unclear directions

-Ambiguity in question terminology

-Inadequate time limits

-Inappropriate level of difficulty

-Poorly constructed test items

Test items are inappropriate for planned test outcomes

-Tests that are too short

-Improper arrangement of items

-Identifiable patterns of answers

-Administration and Scoring

-Nature of the Criterion

-Bias

-Fairness

Test biased towards a certain population

Implies a systematic variation in results

Slope versus Intercept Bias

Rating error - overcome by ranking

Halo effect

A Criterion is the standard against which a test or a test score is evaluated.

*The Criterion needs to be relevant, valid and uncontaminated.

An adequate criterion is relevant, i.e. it is pertinent or applicable to the matter in hand.

Evidence should exist that supports the validity of the criterion

A criterion should be uncontaminated, so it should be based, at least in part, on predictor measures. If not, validation study cannot be taken seriously. There is no formal test for criterion contamination

Age, Culture, Gender

Adjustment to scores - is this fair?

Psychometric techniques for reducing adverse impact of unfairness to some groups

The method developed by Lawshe gauged agreement among raters/judges regarding how essential a particular item is.

i. e. is the skill or knowledge measured by this item:

essential

useful (but not essential)

unnecessary

If test scores are obtained about the same time that the criterion measures are obtained, measures of the relationship between the test scores & the criterion provide evidence of concurrent validity.

Measures of the relationship between test scores & a criterion measure obtained at some future time provide an indication of the predictive validity of the test, that is, how accurately scores on the test predict some criterion measure

Two types of statistical evidence is used:

the validity coefficient and

expectancy data

The Validity coefficient is a correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure.

*The Pearson Correlation coefficient is typically used to determine the validity between the 2 measures.

restriction or inflation of range, a key issue being whether the range of scores employed is appropriate to the objective of the correlational analysis

Cronbach & Gleser cautioned against the establishment of such a rule, it simply should be high enough to result in the identification & differentiation of test-takers with regard to target ability.

Incremental validity is the degree to which an additional predictor explains something about the criterion measure that is not explained by predictors already in use.

Each measure used as a predictor should have criterion-related predictive validity, possess incremental validity & only be included if they demonstrate something not covered by existing predictors

Expectancy data provides information that can be used in evaluating the criterion-related validity of a test.

*Expectancy tables illustrate the likelihood that the test-taker will score within some interval of scores on a criterion measure (e.g. pass/fail)

Taylor-Russell (1939, 1973, 1974) - seven steps to an expectancy table

Naylor-Shine Tables (1965)