Standard Setting

Candidates taking licensure, certification, credentialing, or achievement tests must demonstrate through their score performance that they possess predefined levels of competency. These levels of competency are often defined operationally as "cut scores" on the tests. The process of recommending a cut score is called "standard setting." A minimum cut score can identify any target level of performance. This could be a "minimally competent" or a "minimally advanced" candidate.           

Why Do We Need Standard Setting? 

Important decisions (e.g., whether a student will graduate or not) are often based on the cut scores. These decisions can influence candidates' livelihoods (in the licensure and credentialing fields) and futures (in graduation tests). Therefore, the purpose of a standard-setting study is to make the most accurate and defensible prediction of the cut scores so they are fair to all candidates.  

On the other hand, candidates who do not demonstrate adequate competency (e.g., in a medical practitioner licensure test) should not be given a license to provide services to the public. Doing so would be putting the public at risk from these ill-prepared practitioners. The standard-setting process is a bit of a balancing act: ill-prepared candidates should be treated fairly but also should not be licensed (or, in the case of high school students, graduate).

Methodologies 

Many standard-setting methods developed over the last 30 years have also been used for setting cut scores on high-stakes tests. Most of the methods involve a panel of judges, who are considered experts in the content domain being assessed and must be able to evaluate the knowledge and skills possessed by the target candidate. Judges selected for standard setting have generally utilized one of two methodologies, "candidate-centered" or "test-centered."

In candidate-centered methodology (e.g., the Borderline or Body-of-Work methods), the panel of judges classifies the candidates they know into ability-level groups, including a group consisting of those on the borderline of the demonstrated proficiency. In this method, the cut score is derived from the scores the borderline candidate group achieved on the test. Candidate-centered methods consist of six general steps:

  1. Select the judges.
  2. Define adequate and inadequate levels of the knowledge and skills tested.
  3. Select the sample of test-takers whose skills will be judged.
  4. Obtain the test scores and the judgments about the test-takers you have selected, but do not let the judges know the test-takers' scores.
  5. Divide the test-takers into "qualified" and "unqualified" groups, based on the experts' judgments.
  6. Combine the judgments to choose a cut score.

In the test-centered methodology (e.g., the Bookmark or Angoff methods), judges make decisions about anticipated candidate performance on each item in the test. Test-centered methods consist of five general steps:

  1. Select the judges.
  2. Define "borderline" knowledge and skills.
  3. Train the judges in the use of the method chosen.
  4. Collect judgments.
  5. Combine the judgments to choose a cut score.  

Evaluation of the Standard Setting Procedure  

Since standard setting is a judgmental procedure, different foci of judges produce different standards. One group of judges may set the cut score at one raw score point on the test, while another group of judges may set it at a different raw score point. In order to gather some validity evidence of this process, we collect information about the judges' level of satisfaction with a predefined level of competency; about the judges' training; and about the judges' level of confidence and comfort with the cut score they set, etc. Along with this information, we also try to identify the agreement or disagreement rate among the judges, as well as their consistency across the items on the test. Additionally, in the educational framework, we can compare standards across grade levels to evaluate overall consistency of standards within an assessment program.

Conclusion

No matter which methodology we use for setting the standards, judges' understanding of the predefined level of competency is the most critical component of the whole process. Ultimately, standard setting is a procedure in which we try to ensure fair judgment to the candidates, while protecting the public from ill-prepared individuals who may put them at risk. 

Want to learn more? Connect to this topic within our Research Framework.