Psychological - W2 - Chapter 5 - Reliability (DN) Part 2
This deck covers key concepts related to reliability in psychological testing, including various reliability estimates, item response theory, and measurement errors.
inflation of range/variance
SAMPLING PROCEDURES may impact the variance of either variable in a correlation analysis
OUTCOME
if variance of EITHER variable is INFLATED by sampling procedure then the resulting CC tends to be HIGHER (i.e., giving a false indicator of correlation
(thought to self - is this also a validity issue e.g., false positive)
conversely referred to as RESTRICTION OF RANGE/VARIANCE
if variance of EITHER variable is RESTRICTED by sampling procedure used, then tends to be a LOWER CORRELATION COEFFICIENT (i.e., masking true correlation)
(thought to self - is this also a validity issue e.g., failing to detect - a miss!!!)
p.162
Key Terms
inflation of range/variance
SAMPLING PROCEDURES may impact the variance of either variable in a correlation analysis
OUTCOME
if variance of EITHER variable is INFL...
information function
an IRT TOOL
helps test users to determine the RANGE OVER THETA for which an item is most useful in DISCRIMINATING among groups of testtakers<...
inter-item consistency
the CONSISTENCY or HOMOGENEITY of ALL items on a test
ESTIMATED by techniques such as the SPLIT-HALF RELIABILITY method
the DEGREE of C...
internal consistency estimate of reliability
an ESTIMATE of the RELIABILITY of a test
| - obtained from a MEASURE of INTER-ITEM CONSISTENCY p.152
inter-scorer reliability
An ESTIMATE of the DEGREE of agreement or CONSISTENCY between TWO or more SCORERS on a test.
also referred to as INTER-RATER reliability; OBS...
item characteristic curve (ICC)
graphic representation of the PROBABILISTIC RELATIONSHIP between a person's LEVEL of TRAIT (ability, characteristic) being measured and the PROBABI...
Related Flashcard Decks
Study Tips
- Press F to enter focus mode for distraction-free studying
- Review cards regularly to improve retention
- Try to recall the answer before flipping the card
- Share this deck with friends to study together
| Term | Definition |
|---|---|
inflation of range/variance | SAMPLING PROCEDURES may impact the variance of either variable in a correlation analysis OUTCOME if variance of EITHER variable is INFLATED by sampling procedure then the resulting CC tends to be HIGHER (i.e., giving a false indicator of correlation (thought to self - is this also a validity issue e.g., false positive) conversely referred to as RESTRICTION OF RANGE/VARIANCE if variance of EITHER variable is RESTRICTED by sampling procedure used, then tends to be a LOWER CORRELATION COEFFICIENT (i.e., masking true correlation) (thought to self - is this also a validity issue e.g., failing to detect - a miss!!!) p.162 |
information function | an IRT TOOL helps test users to determine the RANGE OVER THETA for which an item is most useful in DISCRIMINATING among groups of testtakers p. 171 |
inter-item consistency | the CONSISTENCY or HOMOGENEITY of ALL items on a test ESTIMATED by techniques such as the SPLIT-HALF RELIABILITY method the DEGREE of CORRELATION among ALL ITEMS on a scale - p.154 |
internal consistency estimate of reliability | an ESTIMATE of the RELIABILITY of a test | - obtained from a MEASURE of INTER-ITEM CONSISTENCY p.152 |
inter-scorer reliability | An ESTIMATE of the DEGREE of agreement or CONSISTENCY between TWO or more SCORERS on a test. also referred to as INTER-RATER reliability; OBSERVER reliability; JUDGE reliability; SCORER reliability. p.159, 161 |
item characteristic curve (ICC) | graphic representation of the PROBABILISTIC RELATIONSHIP between a person's LEVEL of TRAIT (ability, characteristic) being measured and the PROBABILITY for responding to an item in a PREDICTED way; also known as a CATEGORY RESPONSE CURVE, or, an ITEM TRACE LINE p. 177, 281 |
item response theory (IRT) | another alternative to the true score model a family of theories/methods (well over 100 varieties of IRT models) each model is designed to HANDLE data with CERTAIN ASSUMPTIONS a way of modelling (predicting?) the PROBABILITY that a person with X ability will be able to perform at a LEVEL OF Y. also referred to as LATENT-TRAIT MODELp. p. 166, 168-173 |
item sampling | one source of VARIANCE in the measurement process is the VARIATION among items WITHIN a test, or BETWEEN tests i.e., the way in which a test is CONSTRUCTED is a source of ERROR VARIANCE also CONTENT SAMPLING p. 147 |
Kuder-Richardson formula 20 (KR-20) | a series of EQUATIONS developed by G. F Kuder & M. W. Richardson designed to ESTIMATE the INTER-ITEM CONSISTENCY of tests only appropriate for use on tests with DICHOTOMOUS ITEMS (true/false) p. 155-156, 163 |
latent-trait theory | a synonym for IRT (Item Response Theory) in the academic literature a system of ASSUMPTIONS about measurement includes ASSUMPTION that a TRAIT being measured is UNIDIMENSIONAL go back and check this pg 168 - the extent to which each test item measures the targeted trait also referred to as LATENT-TRAIT MODELp. 168 |
measurement error | all factors associated with the PROCESS of measuring some variable OTHER than the actual variable being measured p.146 |
odd-even reliability | an ESTIMATE of the SPLIT-HALF RELIABILITY of a test | - Splitting a test by assigning odd-numbered items to one half & even-numbered items to the other half of the test p.153 |
parallel forms | when on each FORM of the test, the MEANS & VARIANCES of OBSERVED TEST SCORES are EQUAL .151 |
parallel-forms reliability | an estimate of the consistency of two versions of a test across time an ESTIMATE of the extent to which ITEM SAMPLING & OTHER ERRORS have affected test scores on versions of the SAME test, for which MEANS & VARIANCES of OBSERVED TEST SCORES are EQUAL. (contrast with alternate forms reliability & also coefficient of equivalence) p.151-152 |
polytomous test item | a test item or question with THREE OR MORE ALTERNATIVE RESPONSES where ONLY ONE is scored CORRECT or is CONSISTENT with a TARGETED TRAIT or other CONSTRUCT p. 169 |
power test | a test, usually of achievement or ability has 1) either NO TIME LIMIT or such a long time limit that ALL TESTAKERS can attempt ALL ITEMS 2) some items are SO DIFFICULT that NO TESTTAKER can obtain a PERFECT SCORE (so its isolating the 'power' or 'ability' variable) (contrast with speed test) p.163 |
random error | a source of ERROR when measuring a target variable due to UNPREDICTABLE FLUCTUATIONS & INCONSISITENCIES of OTHER VARIABLES in the measurement process - sometimes referred to as "NOISE" - contrast with systematic error p.146 |
Rasch model | a reference to an IRT MODEL with VERY SPECIFIC ASSUMPTIONS about the UNDERLYING DISTRIBUTION p.169 |
reliability | the proportion of the total variance attributable to TRUE VARIANCE - the GREATER the proportion of TRUE VARIANCE = the GREATER the RELIABILITY of a test - p.157-158 |
reliability coefficient | general term an INDEX of RELIABILITY - or the RATIO of TRUE SCORE VARIANCE to TOTAL SCORE VARIANCE on a test p. 145 |