Psychological - Lecture 2 Validity & Reliability (Catherine) Part 4

7 step procedure was provided

Answer: The table can assist in judging the utility of a test by determining the increase over current procedures

Limitation:

The relationship between predictor and criterion must be linear

It is difficult to identify the cut off for successful vs unsuccessful using the table

No need for linear relationship as uses average criterion scores to compare

Obtaining the difference between the means of the selected & unselected groups to derive an index of what the test is adding to already established procedures

Identifies the utility of a test by determining the increase in average score on some criterion measure

With both tables the validity coefficient used must be one obtained by Concurrent Validation procedures

Cronbach & Gleser's Psychological Tests & Personnel Decision (1957, 1965)

a classification of decision problems

various selection strategies ranging from single-stage to sequential analyses

a quantitative analysis of the relationship between test utility, the selection ratio, costing of testing, expected value of outcomes &

a recommendation that in some instances job requirements be tailored to the applicants abilities instead of the other way around

Construct Validity is a judgment about the appropriateness of inferences drawn from test scores regarding individual standings on a variable called a construct.

*Constructs are observable, presupposed or underlying traits that a test developer may invoke to describe test behaviour or criterion performance

High scorers and low scorers will behave as predicted by the theory

NB Construct validity has been viewed as the unifying concept for all validity evidence

The test simply does not measure the construct

| * The theory is sound, but the statistical procedures or their execution was flawed

The test is homogeneous, measuring a single construct

Test scores increase or decrease as a function of age, the passage of time, or an experimental manipulation as theoretically predicted

Test scores obtained after some event or time (i.e. post-test scores) differ from pre-test scores as theoretically predicted

Test scores obtained by people from distinct groups vary as predicted by the theory

Test scores correlate with scores on other tests in accordance with what would be predicted from a theory that covers the manifestation of the construct in question

Evidence for the construct validity of a particular test may converge from a number of sources, e.g. other tests designed to assess the same or similar construct. Thus, if scores on the test undergoing construct validation tend to correlate highly in the predicted direction with scores on older, more established, already validated tests, this is convergent evidence

A validation coefficient showing little (i.e. a statistically insignificant) relationship between test scores &/or other variables with which scores on the test being construct-validated should NOT theoretically be correlated this provides Discriminant Evidence or discriminant validity

Factor Analysis

If more than half the panelists indicate the item is essential, it has at least some content validity.

Lawshe recommended that if the amount of agreement observed is more than 5% likely to occur by chance, then the item should be eliminated

Negative CVR: fewer than half the panelists indicate essential

zero CVR: exactly half the panelists indicate essential

Positive CVR: More than half, but not all the panelists indicate essential

That errors in measurement exist in all test.

these errors affect both reliability and validity

the test developers goal is to reduce / minimise error

If a test systematically under predicts or over predicts the performance of a particular group with respect to a criterion, then it exhibits intercept bias.

Intercept bias is a term derived from the point where the regression line intersects the Y-Axis

If a test systematically yields significantly different validity coefficients for members of different groups, then it has a slope bias

Slope bias is named as the slope of one group's regression line is different in a statistically significant way from the slope of another group's regression line.

A rating error is a judgment resulting from the intentional or unintentional misuse of a rating scale.

Leniency or generosity error (too generous)

Severity error (too harsh)

Central Tendency error (sticks to the middle)

Halo Effect (high ratings in all things due to raters failure to discriminate)

To obtain consistent results that truly reflect the concepts we are trying to obtain.

Fairness, in a psychometric context is the extent to which a test is used in an impartial, just, and equitable way.

The influence of Culture extends to judgements concerning validity of tests and test items