Psychological - Lecture 2 Validity and Reliability (Catherine)

An instrument has:

Precision if it has fineness of discrimination

Accuracy if it gives the correct value and has no systematic bias

Reliability if the instrument has measurement stability, with no random substantial fluctuations

Validity if it measures what it proports to measure

Test-retest Reliability: Correlating pairs of scores on 2 different administrations of the same test

Internal Consistency Reliability: split-half testing Cronbach (nondichotomous items) Kuder & Richardson (dichotomous items)

Inter-scorer Reliability: The degree of agreement between scorers

Stability over time?

Internal consistency

Test scores are made up of the true score plus error

There is always variability in test scores as a result of error

-Stability over time: central problem is the interpretation of individual scores changes when the test is administered multiple times

The extent to which a psychological test is homogenous (tests one item) or heterogeneous (tests more than one item)

- DASS tests depression, anxiety, & stress is therefore heterogeneous

-Test Construction Item or content sampling

-Test Administration (environment, test-taker & examiner related variables)

-Test Scoring & Interpretation

(Hand scoring or subjective judgements)

-Systematic error: an ambiguously asked question could be interpreted differently by 2 people

-Errors in Item or content sampling

-Random error: e.g. Catherine is morning person, Donna is afternoon person if we both have our exam in the morning, Catherine has an advantage

NB: Can use Alternate Forms to identify source of error and internal consistency for fatigue as source of error variance

-Inconsistent environmental factors (e.g. air con vs no air con)

-Test-taker (individual differences not taken into account like age)

-Examiner related error (fatigue, boredom, etc)

NB: Can use Test-Retest to identify source of error

Hand scoring open to error

Subjective judgements

Computer aided scoring cannot be used for qualitative data

Test-Retest Reliability

Parallel Forms Reliability

Alternate Forms Reliability

Internal Consistency Reliability using

Split-Half Reliability

Chronbach’s alpha

Kuder-Richardson

Inter-Scorer Reliability

Test-Retest Reliability Testing attempts to account for Errors in Test Administration

The Test is taken twice and the results are correlated

It is important to have an appropriate amount of time between tests (this will vary depending on the type of test - e.g. MSE needs 18 months,)

Systematic Changes should not affect the scores (e.g. everyone in a cold room)

Unpredictable changes will affect the correlation (such as?)

A reliable test will be able to sustain greater levels of fluctuation (what does that mean?)

individual differences,

experience,

practice effects,

memory,

fatigue,

motivation.

Parallel Forms or Alternative Forms Reliability Tests attempt to account for errors in Test Construction

In a situation where it is not possible to conduct a Test-Retest Reliability test

In both cases the participant completes two tests

- The aim of both is to minimise error variance

Parallel forms of a test exist when for each form of the test the means and variances of observed test scores are equal

Alternate forms are simply different forms of a test that have been constructed to be parallel. They are designed to be equivalent with regard to content and level of difficulty, but do not meet the same stringent criteria as parallel forms (so means & variances have not been made equivalent)

Because the means and variances have not been made equivalent (as they have in parallel form) it leads to highly ambiguous test confounds: as now have two sources of error: Time and Content, whereas with parallel forms just time as a confound.

Split-half reliability testing can be employed to achieve internal consistency reliability

-Ensure the split is in a meaningful way i.e. not first-last half of test (fatigue effects)

better to do odd-even split

-if its heterogeneous test ensure this is also split in meaningful way

A test administrator can obtain a correlation coefficient of a homogeneous test using a Spearman-Brown Formula

-The Spearman-Brown in effect converts the split-half test into a full test.

It cannot be used for a heterogeneous test!

A test administrator can test the internal consistency reliability of any heterogeneous split-half test using a Chronbach’s Alpha Formula

-The Chronbach’s Alpha is a generalised reliability coefficient for scoring systems that are graded by each item.

(With the DASS we would need a Chronbach’s alpha for each trait measured)

It cannot be used for either a homogeneous or a heterogeneous test But NOT dichotomous answers (yes/no; true/false)

A test administrator can test the internal consistency reliability of any dichotomous split-half test using a Kuder-Richardson Formula

The Kuder-Richardson essentially provides an estimate of all test-retest or split-half coefficients for yes/no or true/false answers

Acceptable reliabilities for Clinical settings is:

r > 0.85 acceptable

Acceptable reliabilities for Research settings is: r > ~0.7 acceptable

Internal Consistency of WAIS: r = 0.887

Internal Consistency of MMPI: r = 0.84

Test-Retest Reliabilities of the WAIS: r = 0.82

Test-Retest Reliabilities of the MMPI: r = 0.74

NB: WAIS test-retest is just outside acceptable limits

the MMPI is suseptable to change over time as its a personality inventory used on clinical patients whoa re more likely to change over time

The Test Administrator would be assessing a tests Internal Consistency

Test-Retest Reliability

Alternate Form Reliability

Inter-scorer Relaibility

Is the test measuring state or trait? (trait is more enduring)

The range of possible responses (ideally 5-7 responses - 0 - 10 is not ideal as people tend to cluster around the middle)

Speeded tests - towards end of test test taker may not have had time to attempt a number of items, does not mean they would have been incorrect, did not get time to answer

Quality of items (need to be clear, concise, homogeneous)

Ensure consistent testing conditions

Reduce Test-Retest time intervals

Longer assessments

Develop a robust scoring plan

Test items for reliability & adapt the measure

Ensure Validity

Internal Validity

External Validity

Test Validity

Confidence in making causal statements about study outcomes

Confidence you can generalise results to people outside of the study

Confidence that what you are measuring truly represents what you think you are measuring

Content Validity

Criterion-Related Validity

Construct Validity

Content Validity

Criterion-Related Validity

Construct Validity

Scrutinise test contents

Comparing Scores on this test to other tests

Perform an analysis of how scores on this test relate to scores on other tests and theories

Face Validity relates to whether the person being assessed believes the test appears to measure what is actually being measured

Construct Validity

indirectly tests some aspect not perceived by the test-taker (e.g. MMPI asks about icecream as part of a personality assessment)

may result in negative consequences such as poor test taker attitude, or disgruntlement

Some tests have low face validity and others have high face validity

Content Validity Scrutinises the test's content

Content validity is concerned with how well does each item on the test measure what it intends to measure?

Tests should capture all aspects of the target behaviour

e.g. for HR test items should directly relate to the job role we are hiring for

-We use a Content Validity Ratio (CVR - Lawshe, 1975)

-We ask N number of experts to rate each item to reflect performance on an item in terms of essential, useful, not necessary

-We remove items based on % of people who state the item is not necessary

If less than 5% say necessary/essential it should be removed

Criterion-Related Validity relates scores obtained on the current test to other test scores or other measures

Criterion-Related Validity is a judgement of how adequately a test score can be used to infer an individual's most probable standing on some measure of interest.

The two varieties of Criterion-Related Validity are:

*Concurrent Validity

*Predictive Validity

Criterion-Related Validity is interested in how well the test item reflect an individual's actual score on the criterion of interest

Concurrent Criterion-Related Validity

-The degree to which the score relates to the criterion measure at that time (measure a new test against a gold standard)

Predictive Criterion-Related Validity

-Degree to which the score relates to a criterion measure in the future (i.e. uses regression to predict a person's future reading ability)

Concurrent Validity is an index of the degree to which a test score is related to some criterion masuree obtained at the same time

Is the criterion:

-Relevant

-Valid & Reliable

-Uncontaminated

We need to ensure the test we are comparing is relevant, valid, reliable & uncontaminated, hence we use gold standard tests rather than just random other tests

By performing a correlation and comparing how well the outcome of the new test compares with the outcome of a well-known, reliable, well-validated test

Predictive Validity is an index of the degree to which a test score predicts some criterion measure (in the future)

By obtaining test scores now, and criterion measure in the future using multiple predictors.

-considering the validity coefficient in the context of corresponding issues including

-incremental validity and expectancy data

NEEDS work - not clear

The Validity coefficient is a correlation that provides a measure of the relationship between test scores and scores on the criterion measure.

Uses more than one predictor

Additional predictors used in ascertaining criterion-related predictive validity should possess incremental validity.

That is the degree to which an additional predictor explains something about the criterion measure that is not explained already by predictors already in use.

Expectancy Data provides useful information to evaluate the criterion-related validity of a test.

Using a score obtained from one test or measure expectancy tables illustrate the likelihood that the test-taker will score within some interval of scores (such as pass or fail).

An expectancy table can be created by a scatterplot according to

An expectancy table shows the relationship between scores on e.g. high school exams and relationship to university grade

Construct Validity is concerned with how well inferences drawn from a test score relate to current theories or knowledge (i.e. constructs)

Evidence of Homogeneity

Evidence of Changes with Age

Evidence of Distinct Groups

Convergent Evidence****

Discriminant Evidence****

Factor Analysis

How uniform the test is for measuring a single concept

Correlate sub-sections with the whole test score

Item analysis

How important is homogeneity?

Construct Validity executes a comprehensive analysis of how scores on the test:

a. relate to other scores and measures

b. can be understood within some theoretical framework for understanding the construct that the test was designed to measure

-Unclear directions

-Ambiguity in question terminology

-Inadequate time limits

-Inappropriate level of difficulty

-Poorly constructed test items

Test items are inappropriate for planned test outcomes

-Tests that are too short

-Improper arrangement of items

-Identifiable patterns of answers

-Administration and Scoring

-Nature of the Criterion

-Bias

-Fairness

Test biased towards a certain population

Implies a systematic variation in results

Slope versus Intercept Bias

Rating error - overcome by ranking

Halo effect

A Criterion is the standard against which a test or a test score is evaluated.

*The Criterion needs to be relevant, valid and uncontaminated.

An adequate criterion is relevant, i.e. it is pertinent or applicable to the matter in hand.

Evidence should exist that supports the validity of the criterion

A criterion should be uncontaminated, so it should be based, at least in part, on predictor measures. If not, validation study cannot be taken seriously. There is no formal test for criterion contamination

Age, Culture, Gender

Adjustment to scores - is this fair?

Psychometric techniques for reducing adverse impact of unfairness to some groups

The method developed by Lawshe gauged agreement among raters/judges regarding how essential a particular item is.

i. e. is the skill or knowledge measured by this item:

essential

useful (but not essential)

unnecessary

If test scores are obtained about the same time that the criterion measures are obtained, measures of the relationship between the test scores & the criterion provide evidence of concurrent validity.

Measures of the relationship between test scores & a criterion measure obtained at some future time provide an indication of the predictive validity of the test, that is, how accurately scores on the test predict some criterion measure

Two types of statistical evidence is used:

the validity coefficient and

expectancy data

The Validity coefficient is a correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure.

*The Pearson Correlation coefficient is typically used to determine the validity between the 2 measures.

restriction or inflation of range, a key issue being whether the range of scores employed is appropriate to the objective of the correlational analysis

Cronbach & Gleser cautioned against the establishment of such a rule, it simply should be high enough to result in the identification & differentiation of test-takers with regard to target ability.

Incremental validity is the degree to which an additional predictor explains something about the criterion measure that is not explained by predictors already in use.

Each measure used as a predictor should have criterion-related predictive validity, possess incremental validity & only be included if they demonstrate something not covered by existing predictors

Expectancy data provides information that can be used in evaluating the criterion-related validity of a test.

*Expectancy tables illustrate the likelihood that the test-taker will score within some interval of scores on a criterion measure (e.g. pass/fail)

Taylor-Russell (1939, 1973, 1974) - seven steps to an expectancy table

Naylor-Shine Tables (1965)

7 step procedure was provided

The table can assist in judging the utility of a test by determining the increase over current procedures

Limitation:

The relationship between predictor and criterion must be linear

It is difficult to identify the cut off for successful vs unsuccessful using the table

No need for linear relationship as uses average criterion scores to compare

Obtaining the difference between the means of the selected & unselected groups to derive an index of what the test is adding to already established procedures

Identifies the utility of a test by determining the increase in average score on some criterion measure

With both tables the validity coefficient used must be one obtained by Concurrent Validation procedures

Cronbach & Gleser's Psychological Tests & Personnel Decision (1957, 1965)

a classification of decision problems

various selection strategies ranging from single-stage to sequential analyses

a quantitative analysis of the relationship between test utility, the selection ratio, costing of testing, expected value of outcomes &

a recommendation that in some instances job requirements be tailored to the applicants abilities instead of the other way around

Construct Validity is a judgment about the appropriateness of inferences drawn from test scores regarding individual standings on a variable called a construct.

*Constructs are observable, presupposed or underlying traits that a test developer may invoke to describe test behaviour or criterion performance

High scorers and low scorers will behave as predicted by the theory

NB Construct validity has been viewed as the unifying concept for all validity evidence

The test simply does not measure the construct

| * The theory is sound, but the statistical procedures or their execution was flawed

The test is homogeneous, measuring a single construct

Test scores increase or decrease as a function of age, the passage of time, or an experimental manipulation as theoretically predicted

Test scores obtained after some event or time (i.e. post-test scores) differ from pre-test scores as theoretically predicted

Test scores obtained by people from distinct groups vary as predicted by the theory

Test scores correlate with scores on other tests in accordance with what would be predicted from a theory that covers the manifestation of the construct in question

Evidence for the construct validity of a particular test may converge from a number of sources, e.g. other tests designed to assess the same or similar construct. Thus, if scores on the test undergoing construct validation tend to correlate highly in the predicted direction with scores on older, more established, already validated tests, this is convergent evidence

A validation coefficient showing little (i.e. a statistically insignificant) relationship between test scores &/or other variables with which scores on the test being construct-validated should NOT theoretically be correlated this provides Discriminant Evidence or discriminant validity

Factor Analysis

If more than half the panelists indicate the item is essential, it has at least some content validity.

Lawshe recommended that if the amount of agreement observed is more than 5% likely to occur by chance, then the item should be eliminated

Negative CVR: fewer than half the panelists indicate essential

zero CVR: exactly half the panelists indicate essential

Positive CVR: More than half, but not all the panelists indicate essential

That errors in measurement exist in all test.

these errors affect both reliability and validity

the test developers goal is to reduce / minimise error

If a test systematically under predicts or over predicts the performance of a particular group with respect to a criterion, then it exhibits intercept bias.

Intercept bias is a term derived from the point where the regression line intersects the Y-Axis

If a test systematically yields significantly different validity coefficients for members of different groups, then it has a slope bias

Slope bias is named as the slope of one group's regression line is different in a statistically significant way from the slope of another group's regression line.

A rating error is a judgment resulting from the intentional or unintentional misuse of a rating scale.

Leniency or generosity error (too generous)

Severity error (too harsh)

Central Tendency error (sticks to the middle)

Halo Effect (high ratings in all things due to raters failure to discriminate)

To obtain consistent results that truly reflect the concepts we are trying to obtain.

Fairness, in a psychometric context is the extent to which a test is used in an impartial, just, and equitable way.

The influence of Culture extends to judgements concerning validity of tests and test items