Psychological Testing: Chapter 8: Test Development

Test Conceptualization
Test Construction
Test Tryout 
Item Analysis
Test Revision

Drafting of items for the test

First draft of the test is then tried out on a group of sample testtakers

When statistical procedures are employed to assist in making judgments about which items are good as they are, which items need to be revised, and which items should be discarded

Analyses of item reliability
Item Validity
Item Discrimination

There ought to be a test designed to measure (____) in a (____) way; stimulus could be anything; review of related literature on existing tests

What is the test designed to Measure?
What is the objective of the test?
Is there a need for this test?

Closely linked to how the test developer defines the construct being measured and how that definition is the same as or different from other tests purporting to measure the same construct

In service of what goal will the test be employed? In what way or ways is the objective of this test the same as or different from other tests with similar goals? What real-world behaviors would be anticipated to correlate with testtaker responses?

Are there any other tests purporting to measure the same thing? In what ways will the new test be better than or different from existing ones? Will there be more compelling evidence for its reliability or validity? Will it be more comprehensive? Will it take less time to administer? In what ways would this test not be better than existing tests?

Who will use this test?
Who will take this test?
What content will the test cover?
How will the test be administered?
What is the ideal format of the test?
Should more than one form of the test be developed?
What special training will be required of test users for administering and interpreting the test?
What types of responses will be required of testtakers?
Who benefits from an administration of this test?
Is there any potential for harm as a result of an administration of this test?
How will meaning be attributed to scores on this test?

Clinicians? Educators? Others? For what purpose or purposes would this test be used?

Who is this test for? Who needs to take it? Who would find it desirable to take it? For what age range of testtakers is the test designed? What reading level is required of a testtaker? What cultural factors might affect the testtaker response?

Why should it cover this content? Is this coverage different from the content coverage of existing tests with the same or similar objectives? How and why is the content area different? To what extent is this content culture-specific?

Individually or in groups? Is it amenable to both group and individual administration? What differences will exist between individual and group administrations of this test? Will the test be designed for or amenable to computer administration? How might differences between versions of the test be reflected in test scores?

On the basis of a cost-benefit analysis, should alternate or parallel forms of this test be created?

What background and qualifications will a prospective user of data derived from an administration of this test need to have? What restrictions, if any, should be placed on distributors of the test and on the test’s usage

What kind of disability might preclude someone from being able to take this test? What adaptations or accommodations are recommended for persons with disabilities?

What would the testtaker learn, or how might the testtaker benefit, from an administration of this test? What would the test user learn, or how might the test user benefit? What social benefit, if any, derives from an administration of this test?

What safeguards are built into the recommended testing procedure to prevent any sort of harm to any of the parties involved in the use of this test?

Will a testtaker’s score be compared to others taking the test at the same time? To others in a criterion group? Will the test evaluate masters of a particular content area?

An item for which high scorers on the test respond correctly; low scorers on the test tend to respond to that same item incorrectly

High scorers on the test get a particular item right whereas low scorers on the test get that same item wrong; each item should address the issue of whether the testaker has met certain criteria

Refers to the preliminary reserach surrounding the creation of a prototype of the test; test items may be piloted to evaluate whether they should be included in the final form of the instrument; May involve open-ended interviews with research subjects believed for some reason (perhaps on the basis of an existing test); developer attempts to determine how best to measure a targeted construct

Entails the Creation
Revision
Deletion of many test items
Literature Reviews
Experimentation
Related activities

Assignment of numbers according to rules; defined as the process of setting rules for assigning numbers in measurement; process by which a measuring device is designed and calibrated and by which numbers (or other indices)-scale values- are designed to different amounts of the trait, attribute, or characteristic being measured

If the Testtaker's test performance as a function of age is of critical interest

If the testtaker's test performance as a function of grade is of critical interest

If all raw scores on the test are to be transformed into scores that can range from 1 to 9

Unidimentional vs. Multidimensional

| Comparative vs. Categorical

Defined as a grouping of words, statements, or symbols on which judgments of the strength of a particular trait, attitude, or emotion are indicated by a testtaker; can be used to record judgments of oneself, other, experiences, or objects, and that they can take several forms

When final test score is obtained by summing the ratings across all the items

used extensively in psychology, usually to scale attitudes; relatively easy to construct

Testtakers are presented with pairs of stimuli which they are asked to compare; select one of the stimuli according to some rule; the rule that they agree more with one statement than the other, or the rule that they find one stimulus more appealing than the other;

Entails judgments of a stimulus in comparison with every other stimulus on the scale

Scaling system that relies on sorting; stimuli are place into one of two or more alternative categories that differ quantitatively with respect to some continuum

Another scaling method that yields ordinal-level measures; items range sequentially from weaker to stronger expressions of the attitude, belief, or feeling being measured; all respondents who agree with the stronger statements of the attitude will also agree with milder statements

Item-analysis procedure and approach to test development that involves a graphic mapping of a testtaker's responses; Objective for the developer of a measure of attitudes is to obtain an arrangement of items wherein endorsement of one item automatically connotes endorsement of less extreme positions

Collect a reasonably large number of statements reflecting positive and negative attitudes towards a topic are collected
Judges or experts evaluate each statement in terms of how strongly it indicates that the topic is justified. Each judge is instructed to rate each statement on a scale as if the scale were interval in nature
Mean and a standard deviation of the judges' ratings are calculated for each statement
Items are selected for inclusion in the final scale based on several criteria, including (a) the degree to which the item contributes to a comprehensive measurement of the variable in question (b) the test developer's degree of confidence that the items have indeed been sorted into equal intervals
Scale is now ready for administration; the way the scale is used depends on the objectives of a test situation

Variables being measured
Group for whom the test is intended
Preferences of the test developer

What range of content should the items cover?
Which of the many different types of item formats should be employed?
How many items should be written in total and for each content area covered?

Reservior or well from which test items will or will not be drawn for the final version of the test

Include variables such as the form, plan, structure, arrangement, and layout of individual test items

Selected-Response Format

| Constructed-Response Format

Require testtakers to select a response from a set of alternative responses

Require the testtakers to supply or to create the correct answer, not merely to select it

Multiple Choice
Matching
True or False

Stem
Correct Alternative or option
Several incorrect alternatives or options variously referred to as distractors or foils

Has one correct alternative
Has grammatically parallel alternatives
Has alternatives of similar length
Has alternatives that fit grammatically with the stem
Includes as much of the item as possible in the stem to avoid unnecessary repetition
Avoids ridiculous distractors

Testtaker is presented with two columns: Premises on the left and responses to the right;

Multiple-choice item that contains only two possible responses

The most familiar binary-choice item; type of selected-response item which takes the form of a sentence that requires the testtaker to indicate whether the statement is or is not a fact

Contains a single idea, is not excessively long, and is not subject to debate; correct response must undoubtedly be one of the two coices

Requires the examinee to provide a word or phrase that completes a sentence; also known as Short-Answer Item

Should be worded so that the correct answer is specific; Should be written clearly enough that the testtaker can respond succinctly (with a short answer)

Useful when the test developer wants the examinee to demonstrate a depth of knowledge about a single topic; permits restating of learned material and allows for the creative integration and expression of the material in the testtaker's own words; subjective and inter-scorer differences

Relatively large and accessible collection of test questions; advantage is accessibility to a large number of test items conveniently classified by subject area, item statistics, or other variables

Technique with the ability to individualize testing; ability of the computer to tailor the content and order of presentation of test items on the basis of responses to previous items

Refers to an interactive, computer-administered testtaking procedure wherein items presented to the testtaker are absed in part on the testtaker's performance on previous items; tends to reduce floor effects and ceiling effects

Refers to the diminished utility of an assessment tool for distinguishing testtakers at the low end of the ability, trait, or other attribute being measured

Refers to the diminished utility of an assessment tool for distinguishing testtakers at the high end of the ability, trait, or other attribute being measured

Employs testtaker responses which earn credit toward placement in a particular class or category with other testtakers whose pattern of responses is presumably similar in some way; used by dome diagnostic systems wherein individuals must exhibit a certain number of symptoms to qualify for a specific diagnosis

Comparing a testtaker's score on one scale within a test to another scale within that same test

EPPS designed to measure the relative strength of different psychological needs

Cross Validation
Co-Validation
Quality Assurance During Test Revision

Stimulus materials look dated and current testtakers cannot relate to them.
Verbal Content of the test, including the administration instructions and the test items, contains dated volcabulary that is not readily understood by current testtakers
As popular culture changes and words take on new meanings, certain words or expressions in the test items or directions may be perceived as inappropriate or even offensive to a particular group and must therefore be changed.
Test norms are no longer adequate as a result of age-related shifts in the abilities measured over time, and so an age extension of the norms (upward, downward, or in both directions) is necessary
The reliability or the validity of the test, as well as the effectiveness of individual test items, can be significantly improved by a revision
The theory on which the test was originally based has been improved significantly, and these changes should be reflected in the design and content of the test.

Refers to the revalidation of a test on a sample of testtakers other than those on whom test performance was originally found to be a valid predictor of some criterion

The decrease in item validities that inevitably occurs after cross-validation of findings; expected and viewed as integral to the test development process; infinitely preferable to a scenariou wherein high item validities are published in a test manual as a result of inappropriately using the identical sample of testtakers for test standardization and cross-validation of findings

Should outline the test development procedures used

| Reliability information, including test-retest validity and Internal consistency estimates

Defined as a test validation process conducted on two or more tests using the sample of testtakers

Process that occurs when co-validation is used in conjunction with the creation of norms or the revision of existing norms

Test protocol scored by a highly authoritative scorer that is designed as a model for schoring and a mechanism for resolving scoring discrepancies

A discrepancy between scoring in an anchor protocol and the scoring of another protocol

Evaluating existing tests for the purpose of mapping test revisions
Determining measurement equivalence across testtaker populations
Developing item banks

Help test developers evaluate how well an individual item (or entire test) is working to measure different levels of the underlying construct
Can be used to weed out uninformative questions
Eliminate redundant items to tailor an instrument to provide high information (Precision)

Phenomenon wherein an item functions differently in one group of testtakers as compaered to another group of testtakers known to have the same (or similar) level of the underlying Trait

A process by which test developers scrutinize group-by-group item response curves, looking for DIF Items; used to evaluate the effect of different test administration procedures and item ordering effects

Items that respondents from different groups at the same level of the underlying traid have different probabilities of endorsing as a function of their group membership