Evaluation
• Process that can justify that what we do as nurses and as a nurse educators make a value-added difference in the care we provide
• It is a systematic process by which the worth or value of something- in this case, teaching and learning – is judged
• It is the final component of the process and serve as the critical bridge at the end of one cycle that guides direction of the next cycle
• As an after thought is, at best, a poor idea and, at worst, a dangerous one
DETERMINING THE FOCUS OF EVALUATION
Evaluation focus on five basic components:
• AUDIENCE- comprises the person or groups for whom the evaluation is being conducted
• PURPOSE- to decide whether to continue a particular education program or to determine the effectiveness of the teaching process
• QUESTION- to be asked in the evaluation are directly related to the purpose for conducting the evaluation, are specific and are measurable
• SCOPE- can be considered an answer to the question
• RESOURCES- include time, expertise, personnel, materials, equipment, and facilities
EVALUATION MODELS
• PROCESS ( FORMATIVE) EVALUATION
its purpose is to make adjustments in an educational activity as soon as they are needed, whether those adjustments be in personnel, materials, facilities, learning objectives or even ones attitude . This is ongoing evaluation helps the nurse anticipate and prevent problems as they arise.
o CONTENT EVALUATION
The purpose is to determine whether learners have acquired the knowledge or skills taught during the learning experience. Take place immediately after the learning experience.
o OUTCOME (SUMMATIVE) EVALUATION
Is to determine the effects or outcomes of teaching efforts, it also referred to as SUMMATIVE EVALUATION because its meant is to sum what happened as a result of education. It measures changes occurring as a result of teaching and learning. Focus on measuring more long term change that persists after the learning experience.
o IMPACT EVALUATION
The purpose is to obtain information that will help decide whether continuing an educational activity is worth cost. Focus on a course of goal.
o PROGRAM EVALUATION
The purpose is to determine the extent to which all activity for an entire department or program over specified of time meet or exceed goals originally established. The scope of program is broad, generally focusing on overall goals rather on specific objectives.
Evaluation of learning book from teaching strategies
Tests- an art as well as a science and cannot be fully taught in a single chapter
- may consist of a single type of question or a combinations of types, as long as the objectives are being measured at the desired levels of learning
TYPES OF TESST QUESTION
Multiple Choice QUESTIONS
- nursing examination are often written in this types of question for such reasons of
1. challenging to create
2. easy to score and can be scored by the computer
3. NLE are multiple choice therefore educator wants the learner to be familiar in this type of questions
Two parts of multiple choice question
1. STEM – the question itself
- should be short as possible while still conveying the ideas clearly
- Negative terms should be avoided because it tends to make questions more confusing
2. OPTIONS/ DISTRACTERS - the possible answer or solution
- should be realistic
Few rules govern in writing of options
1. grammatically consistent with the stem
2. use good style and avoided giving clues
3. options should be fairly short and about the same length
4. if the answer is written longer, increase the length of the distracters as well
5. options should be place in logical order
6. avoid the use of qualifying terms, such as always, sometimes, usually and never
7. be sure to alter the positions of the correct answers
TRUE- FALSE QUESTIONS
- are designed to tests learner’s ability to identify the correctness of statements of fact or principle
- the weakness is that the learner has a 50/50 chance of guessing the right answer
Variations of true- false question
1. ask the learner to give the rationale for why the item is true or false
2. ask the learner to rewrite false statement to make the true
From Whom or What to Collect Data
WHOM?
• directly from the individuals whose behavior or knowledge is being evaluated
• from surrogates or representatives of these individuals
• from documentation or databases already created
WHAT?
Pre existing databases should never be used as the only source of evaluative data unless they were created for the purpose of that evaluation.
Eventhough these data were collected for different purpose, they may be helpful for providing additional information to the primary audience for the evaluation.
Data already in existence generally are less expensive to obtain than are original data.
The Decision whether to use pre existing data depends on whether they were collected from people of interest in the current evaluation and whether they are consistent with operational definitions used in the current evaluation.
How, When, and Where to Collect Data.
HOW?
Methods for how data can be collected include the following:
• Observation
• Interview
• Questionnaire or written examination
• Record review
• Secondary analysis of existing databases.
Which method is selected depends, first, on the type of data being collected and, second, on available resources. Whenever possible, data should be collected using more than one method. Using multiple methods will provide the evaluator, and consequently the primary audience, with more than could be accomplished using a single method.
WHEN?
The timing of data collection, or when data collection takes place,
has already been addressed both in discussion of different types of evaluation and in descriptions of evaluation design structures.
1. Process evaluation------generally occurs during and immediately after an educational activity.
2. Content evaluation------takes place immediately after completion of education.
3. Outcome evaluation-----occurs some time after completion of education, after learners have returned to the setting where they are expected to use new knowledge or perform a new skill.
4. Impact evaluation------generally is conducted from weeks to years after the educational program being evaluated because the purpose of impact evaluation is to determine what change has occurred within the community or institution as a whole as a result of an educational program.
WHERE?
An appropriate setting for conducting a content evaluation may be in the classroom or skill laboratory where learners have just completed class instruction or training.
An outcome evaluation to determine whether training has improved the nurse’s ability to perform skill with patients on the nursing unit, however, requires that data collection – in this case, observation of the nurse’s performance – be conducted on the nursing unit.
an outcome evaluation to determine whether discharge teaching in the hospital enabled the patient to provide self-care at home requires that data collection, or observation of the patient’s performance, be conducted in the home.
Who Collects Data.
WHO?
Combining the role of evaluator is one appropriate method for conducting a process evaluation because evaluative data are integral to the teaching-learning process.
Inviting another educator or a patient representative to observe a class can provide additional data from the perspective of someone who does not have to divide his or her attention between teaching and evaluating. This section, and perhaps less biased, input can strength legitimacy and usefulness of evaluation results.
EVALUATION INSTRUMENT
This chapter is intended to present keypoints to consider in selection, modification, or construction of evaluation instruments.
The initial step in instrument selection is to conduct a literature search for evaluations similar to the evaluation being planned. A helpful place to begin is with the same journals listed earlier in this chapter. Instruments that have used in more than one study should be given preference over an instrument developed for a single use, because instruments used multiple times generally have been more thoroughly tested for reliability and validity.
First, the instrument must measure the performance being evaluated exactly as that performance has been operationally defined for the evaluation.
Second, an appropriate instrument should have documented evidence of its reliability and validity with individuals who are closely matched as possible with the people from whom you will be collecting data.
The evaluation instrument most likely to require modification from an existing tool or development of an entirely new instrument is a cognitive test.
The primary reason for constructing such a test is to be comprehensive and relevant and to fairly test the learner’s knowledge of content covered.
Use of a test blueprint is one of the most useful methods for ensuring comprehensiveness and relevance of test questions because the blueprint enables the evaluator to be certain that each area of course content is included in the test and that content areas emphasized during instruction are similarly emphasized during test.
BARRIERS TO EVALUATION
Barriers to conducting an evaluation can be classified into three broad categories:
1. Lack of Clarity. Lack of clarity most often results from an unclear, unstated, or ill-defined evaluation focus.
2. Lack of Ability. Lack of ability to conduct an evaluation most often results from insufficient knowledge of how to conduct the evaluation or insufficient or inaccessible resources needed to conduct the evaluation.
3.Fear of Punishment or Loss of Self-Esteem. Evaluation may be perceived as judgment of personal worth. Individuals being evaluated may fear that anything less than a perfect performance will result in punishment of that their mistakes will be seen as evidence that they are somehow unworthy or incompetent as human beings.
STEPS ON HOW TO OVERCOME THE BARRIERS
The first step in overcoming this barrier is to realize that the potential for its existence may be close to 100%. Individuals whose performance or knowledge is being evaluated are not likely to say overtly that evaluation represents a threat to them. Rather, they are far more likely to demonstrate self-protective behaviors or attitudes that can range from failure to attend a class that has a post-test, to providing socially desirable answer on a questionnaire, to responding with hostility to evaluation questions.
The second step in overcoming the barrier of fear or threat is to remember that “the person is more important than the performance of the product” (Narrow, 1979, p. 185). If the purpose of an evaluation is to facilitate better learning, as in process evaluation, focus on the process.
A third step in overcoming the fear of punishment or threatening loss of self-esteem is to point out achievements, if they exist, or to continue to encourage effort if learning has not been achieved. Give praise honestly, focusing on the task at hand.
Finally, and perhaps most importantly, use communication of information to prevent or minimize fear. Lack of clarity exists as a barrier for those who are subjects of an evaluation as much as for those who will conduct the evaluation. If learners or educators know and understand the focus of an evaluation, they may be less fearful than if such information is left to their imaginations. Remember that failure to provide certain information may unethical or even illegal.
CONDUCTING THE EVALUATION
To conduct an evaluation means to implement the evaluation design by using the instruments chosen or developed according to the methods selected. How smoothly an evaluation is implemented depends primarily on how carefully and thoroughly the evaluation was planned. Planning is not a complete guarantee of success, however.
3 methods to minimize the effects of unexpected events that occur when carrying out an evaluation are to:
(1) conduct a pilot test first
(2) include “extra” time
(3) keep a sense of humor.
ANALYZING AND INTERPRETING DATA COLLECTED
The purposes for conducting data analysis are
(1) to organize data so that they can provide meaningful information
(2) to provide answers to evaluation questions.
Data and information are not synonymous terms. That is, a mass of numbers or a mass of comments does not become information until it has been organized into coherent tables, graphs, or categories that are relevant to the purpose for conducting the evaluation.
Data can be qualitative or quantitative
Qualitative Data- all qualitative data are at the nominal level of measurement meaning they are described in terms of categories such as health focused versus illness focused
Quantitative Data can be:
• Nominal
• Ordinal
• Interval
• Ratio level of measurement
Data also can be described as continuous or discrete.
Examples of Continuous Data:
1. Age
2. Level of Anxiety
Examples of Discrete Data:
1. Gender
2. Diagnosis
REPORTING EVALUATION RESULTS
Following a few guidelines when planning the likelihood that results of the evaluation will be reported to the appropriate individuals or groups, in a timely manner, and in usable form:
• Be audience focused
• Stick to the evaluation purpose
• Stick to the data
Sunday, August 2, 2009
Clinical teaching
What is classroom assessment?
Classroom assessment is both a teaching approach and a set of techniques. The approach is that the more you know about what and how students are learning, the better you can plan learning activities to structure your teaching. The techniques are mostly simple, non-graded, anonymous, in-class activities that give both you and your students useful feedback on the teaching-learning process.
How is classroom assessment different?
Classroom assessment differs from tests and other forms of student assessment in that it is aimed at course improvement, rather than at assigning grades. The primary goal is to better understand your students' learning and so to improve your teaching.
How do I use Classroom Assessment Techniques?
• Decide what you want to learn from a classroom assessment.
• Choose a Classroom Assessment Technique (CAT) that provides this feedback, is consistent with your teaching style, and can be easily implemented in your class.
• Explain the purpose of the activity to students, then conduct it.
• After class, review the results and decide what changes, if any, to make.
• Let your students know what you learned from the CAT and how you will use this information.
Why should I use CATs?
For faculty, more frequent use of CATs can:
• Provide short-term feedback about the day-to-day learning and teaching process at a time when it is still possible to make mid-course corrections.
• Provide useful information about student learning with a much lower investment of time compared to tests, papers, and other traditional means of learning assessment.
• Help to foster good rapport with students and increase the efficacy of teaching and learning.
• Encourage the view that teaching is a formative process that evolves over time with feedback.
For students, more frequent use of CATs can:
• Help them become better monitors of their own learning.
• Help break down feelings of anonymity, especially in larger courses.
• Point out the need to alter study skills.
• Provide concrete evidence that the instructor cares about learning.
Selected CATs for getting feedback on student learning and response to teaching[1]
Name: Description: What to do with the data: Time required:
Minute paper[2]
During the last few minutes of the class period, ask students to answer on a half-sheet of paper: "What is the most important point you learned today?"; and, "What point remains least clear to you?". The purpose is to elicit data about students' comprehension of a particular class session. Review responses and note any useful comments. During the next class periods emphasize the issues illuminated by your students' comments. Prep: Low
In class: Low
Analysis: Low
Chain Notes Students pass around an envelope on which the teacher has written one question about the class. When the envelope reaches a student he/she spends a moment to respond to the question and then places the response in the envelope. Go through the student responses and determine the best criteria for categorizing the data with the goal of detecting response patterns. Discussing the patterns of responses with students can lead to better teaching and learning. Prep: Low
In class: Low
Analysis: Low
Memory matrix Students fill in cells of a two-dimensional diagram for which instructor has provided labels. For example, in a music course, labels might consist of periods (Baroque, Classical) by countries (Germany, France, Britain); students enter composers in cells to demonstrate their ability to remember and classify key concepts. Tally the numbers of correct and incorrect responses in each cell. Analyze differences both between and among the cells. Look for patterns among the incorrect responses and decide what might be the cause(s). Prep: Med
In class: Med
Analysis: Med
Directed paraphrasing Ask students to write a layman’s "translation" of something they have just learned -- geared to a specified individual or audience -- to assess their ability to comprehend and transfer concepts. Categorize student responses according to characteristics you feel are important. Analyze the responses both within and across categories, noting ways you could address student needs. Prep: Low
In class: Med
Analysis: Med
One-sentence summary Students summarize knowledge of a topic by constructing a single sentence that answers the questions "Who does what to whom, when, where, how, and why?" The purpose is to require students to select only the defining features of an idea. Evaluate the quality of each summary quickly and holistically. Note whether students have identified the essential concepts of the class topic and their interrelationships. Share your observations with your students. Prep: Low
In class: Med
Analysis: Med
Exam Evaluations Select a type of test that you are likely to give more than once or that has a significant impact on student performance. Create a few questions that evaluate the quality of the test. Add these questions to the exam or administer a separate, follow-up evaluation. Try to distinguish student comments that address the fairness of your grading from those that address the fairness of the test as an assessment instrument. Respond to the general ideas represented by student comments. Prep: Low
In class: Low
Analysis: Med
Application cards After teaching about an important theory, principle, or procedure, ask students to write down at least one real-world application for what they have just learned to determine how well they can transfer their learning. Quickly read once through the applications and categorize them according to their quality. Pick out a broad range of examples and present them to the class. Prep: Low
In class: Low
Analysis: Med
Student- generated test questions Allow students to write test questions and model answers for specified topics, in a format consistent with course exams. This will give students the opportunity to evaluate the course topics, reflect on what they understand, and what are good test items. Make a rough tally of the questions your students propose and the topics that they cover. Evaluate the questions and use the goods ones as prompts for discussion. You may also want to revise the questions and use them on the upcoming exam. Prep: Med
In class: High
Analysis: High
(may be homework)
[1] Details on these and others available from Angelo & Cross, Classroom Assessment techniques, 1993.
[2] The Bureau of Evaluative Studies and Testing (BEST) can administer the Minute Paper electronically.
Published Resources:
Angelo, T.A. & Cross, P.K. (1993). Classroom Assessment Techniques (2nd ed.). San Francisco: Jossey-Bass.
Davis, B.G. (1993). Tools for Teaching. San Francisco: Jossey-Bass.
Welcome to Classroom Assessment
Classroom assessments can include a wide range of options -- from recording anecdotal notes while observing a student to administering standardized tests. Regardless of the form, effective classroom assessments provide the feedback and motivation that students need to excel.
This course is divided into 2 parts, each containing 3 lessons. Part I focuses on basic concepts and common assessments such as multiple choice and essay questions. Part II includes performance assessment, classroom interactions, and attitude surveys. Throughout the lessons, resources and techniques for integrating technology are included.
This course can be used as a "self-study" or it can be used for Continuing Education Units (if your district has a facilitator). Each lesson contains sample assessments along with multiple activities that reinforce the concepts. Before you take the course, please fill out the Survey to help gauge your experience level. At the end of the course, a similar survey will be available.
Overview
Selected response assessment items (also referred to as objective assessments) include options such as multiple choice, matching, and true/false questions. These question types can be very effective and efficient methods for measuring students’ knowledge and reasoning. Because many of the standardized tests are based heavily on multiple choice questions, teachers should be skilled at developing effective objective assessment items. In addition, teachers should be able to construct quizzes that target higher level thinking skills (consistent with the application, analysis, and synthesis levels of Bloom’s taxonomy), and they should evaluate their instruments by conducting item analyses.
A. Formative vs. Summative Assessments
Classroom assessments can include a wide range of options -- from recording anecdotal notes while observing a student to administering standardized tests. The options can be roughly divided into two categories -- formative assessments and summative assessments.
Formative assessments are on-going assessments, reviews, and observations in a classroom . Teachers use formative assessment to improve instructional methods and student feedback throughout the teaching and learning process. For example, if a teacher observes that some students do not grasp a concept, she or he can design a review activity or use a different instructional strategy. Likewise, students can monitor their progress with periodic quizzes and performance tasks. The results of formative assessments are used to modify and validate instruction.
Summative assessments are typically used to evaluate the effectiveness of instructional programs and services at the end of an academic year or at a pre-determined time. The goal of summative assessments is to make a judgment of student competency -- after an instructional phase is complete. For example, in Florida, the FCAT is administered once a year -- it is a summative assessment to determine each student's ability at pre-determined points in time. Summative evaluations are used to determine if students have mastered specific competencies and to identify instructional areas that need additional attention.
The following table highlights some formative and summative assessments that are common in K12 schools.
Formative Assessments Summative Assessments
Anecdotal records Final exams
Quizzes and essays Statewide tests (FCAT)
Diagnostic tests National tests
Lab reports Entrance exams (SAT and ACT)
B. Writing Selected Response Assessment Items
Selected response (objective) assessment items are very efficient – once the items are created, you can assess and score a great deal of content rather quickly. Note that the term objective refers to the fact that each question has a right and wrong answer and that they can be impartially scored. In fact, the scoring can be automated if you have access to an optical scanner for scoring paper tests or a computer for computerized tests. However, the construction of these “objective” items might well include subjective input by the teacher/creator.
Before you write the assessment items, you should create a blueprint that outlines the content areas and the cognitive skills you are targeting. One way to do this is to list your instructional objectives, along with the corresponding cognitive level. For example, the following table has four different objectives and the corresponding levels of assessment (relative to Bloom's taxonomy). For each objective, five assessment items will be written, some at Level I and some at Level II. This approach helps to ensure that all objectives are covered and that several higher level thinking skills are included in the assessment.
Objective Number of Items at Level I
(Bloom's Taxonomy) Number of Items at Level II
(Blooms' Taxonomy)
1 2 3
2 3 2
3 1 4
4 4 1
After you have determined how many items you need for each level, you can begin writing the assessments. There are several forms of selected response assessments, including multiple choice, matching, and true/false. Regardless of the form you select, be sure the items are clearly worded at the appropriate reading level, do not include unintentional clues, and are not ambiguous. The validity of your test will suffer tremendously if the students can’t comprehend or read the questions! This section includes a few guidelines for constructing objective assessment items, along with examples and non-examples.
Multiple Choice
Multiple choice questions consist of a stem (question or statement) with several answer choices (distractors). For each of the following guidelines, click the buttons to view an Example or Non-Example.
• All answer choices should be plausible and homogeneous.
o Example
o Non-Example
• Answer choices should be similar in length and grammatical form.
o Example
o Non-Example
• List answer choices in logical (alphabetical or numerical) order.
o Example
o Non-Example
• Avoid using "All of the Above" options.
o Example
o Non-Example
Matching
Matching items consist of two lists of words, phrases, or images (often referred to as stems and responses). Students review the list of stems and match each with a word, phrase, or image from the list of responses. For each of the following guidelines, click the buttons to view an Example or Non-Example.
• Answer choices should be short, homogeneous and arranged in logical order.
o Example
o Non-Example
• Responses should be plausible and similar in length and grammatical form.
o Example
o Non-Example
• Include more response options than stems.
o Example
o Non-Example
• As a general rule, the stems should be longer and the responses should be shorter.
o Example
o Non-Example
True/False
True/false questions can appear to be easier to write; however, it is difficult to write effective true/false questions. Also, the reliability of T/F questions is not generally very high because of the high possibility of guessing. In most cases, T/F questions are not recommended.
• Statements should be completely true or completely false.
o Example
o Non-Example
• Use simple, easy-to-follow statements.
o Example
o Non-Example
• Avoid using negatives -- especially double negatives.
o Example
o Non-Example
• Avoid absolutes such as "always; never."
o Example
o Non-Example
B. Essay Questions (Short and Extended Response)
Essay questions are a more complex version of constructed response assessments. With essay questions, there is one general question or proposition, and the student is asked to respond in writing. This type of assessment is very powerful -- it allows the students to express themselves and demonstrate their reasoning related to a topic. Essay questions often demand the use of higher level thinking skills, such as analysis, synthesis, and evaluation.
Essay questions may appear to be easier to write than multiple choice and other question types, but writing effective essay questions requires a great deal of thought and planning. If an essay question is vague, it will be much more difficult for the students to answer and much more difficult for the instructor to score. Well-written essay questions have the following features:
• They specify how the students should respond.
o Example
o Non-Example
• They provide information about the value/weight of the question and how it will be scored.
o Example
o Non-Example
• They emphasize higher-level thinking skills.
o Example
o Non-Example
Essay questions are used both as formative assessments (in classrooms) and summative assessments (on standardized tests). There are 2 major categories of essay questions -- short response (also referred to as restricted or brief ) and extended response.
Short Response
Short response questions are more focused and constrained than extended response questions. For example, a short response might ask a student to "write an example," "list three reasons," or "compare and contrast two techniques." The short response items on the Florida assessment (FCAT) are designed to take about 5 minutes to complete and the student is allowed up to 8 lines for each answer. The short responses are scored using a 2-point scoring rubric. A complete and correct answer is worth 2 points. A partial answer is worth 1 point.
Sample Short Response Question
(10th Grade Reading)
How are the scrub jay and the mockingbird different? Support your answer with details and information from the article.
Extended Response
Extended responses can be much longer and complex then short responses, but students should be encouraged to remain focused and organized. On the FCAT, students have 14 lines for each answer to an extended response item, and they are advised to allow approximately 10-15 minutes to complete each item. The FCAT extended responses are scored using a 4-point scoring rubric. A complete and correct answer is worth 4 points. A partial answer is worth 1, 2, or 3 points.
Sample Extended Response Question
(5th Grade Science)
Robert is designing a demonstration to display at his school’s science fair. He will show how changing the position of a fulcrum on a lever changes the amount of force needed to lift an object. To do this, Robert will use a piece of wood for a lever and a block of wood to act as a fulcrum. He plans to move the fulcrum to different places on the lever to see how its placement affects the force needed to lift an object.
Part A Identify at least two other actions that would make Robert’s demonstration better.
Part B Explain why each action would improve the demonstration.
A. Fill-in-the-Blank Items
The simplest forms of constructed response questions are fill-in-the-blank or short answer questions. For example, the question may take one of the following forms:
1. Who was the 16th president of the United States?
2. The 16th president of the United States was ___________________.
These assessments are relatively easy to construct, yet they have the potential to test recall, rather than simply recognition. They also control for guessing, which can be a major factor, especially for T/F or multiple choice questions.
When creating short answer items, make sure the question is clear and there is a single, correct answer. Here are a few guidelines, along with examples and non-examples
• Ask a direct question that has a definitive answer.
o Example
o Non-Example
• If using fill-in-the blank, use only one blank per item.
o Example
o Non-Example
• If using fill-in-the blank, place the blank near the end of the sentence.
o Example
o Non-Example
Although constructed response assessments can more easily demand higher levels of thinking, they are more difficult to score. For example, scantrons (optical grade scanners) cannot score this type of assessment, and computer-based scoring is difficult because you must include all synonyms and acceptable answers. For example, all of the following might be acceptable answers to the sample question: "Who was the 16th president of the United States?" Abraham Lincoln; Abe Lincoln; Lincoln; President Lincoln; Honest Abe; the Railsplitter. You might also want to accept common mis-spellings such as Abrahem or Lencoln (depending on the objective).
Classroom assessment is both a teaching approach and a set of techniques. The approach is that the more you know about what and how students are learning, the better you can plan learning activities to structure your teaching. The techniques are mostly simple, non-graded, anonymous, in-class activities that give both you and your students useful feedback on the teaching-learning process.
How is classroom assessment different?
Classroom assessment differs from tests and other forms of student assessment in that it is aimed at course improvement, rather than at assigning grades. The primary goal is to better understand your students' learning and so to improve your teaching.
How do I use Classroom Assessment Techniques?
• Decide what you want to learn from a classroom assessment.
• Choose a Classroom Assessment Technique (CAT) that provides this feedback, is consistent with your teaching style, and can be easily implemented in your class.
• Explain the purpose of the activity to students, then conduct it.
• After class, review the results and decide what changes, if any, to make.
• Let your students know what you learned from the CAT and how you will use this information.
Why should I use CATs?
For faculty, more frequent use of CATs can:
• Provide short-term feedback about the day-to-day learning and teaching process at a time when it is still possible to make mid-course corrections.
• Provide useful information about student learning with a much lower investment of time compared to tests, papers, and other traditional means of learning assessment.
• Help to foster good rapport with students and increase the efficacy of teaching and learning.
• Encourage the view that teaching is a formative process that evolves over time with feedback.
For students, more frequent use of CATs can:
• Help them become better monitors of their own learning.
• Help break down feelings of anonymity, especially in larger courses.
• Point out the need to alter study skills.
• Provide concrete evidence that the instructor cares about learning.
Selected CATs for getting feedback on student learning and response to teaching[1]
Name: Description: What to do with the data: Time required:
Minute paper[2]
During the last few minutes of the class period, ask students to answer on a half-sheet of paper: "What is the most important point you learned today?"; and, "What point remains least clear to you?". The purpose is to elicit data about students' comprehension of a particular class session. Review responses and note any useful comments. During the next class periods emphasize the issues illuminated by your students' comments. Prep: Low
In class: Low
Analysis: Low
Chain Notes Students pass around an envelope on which the teacher has written one question about the class. When the envelope reaches a student he/she spends a moment to respond to the question and then places the response in the envelope. Go through the student responses and determine the best criteria for categorizing the data with the goal of detecting response patterns. Discussing the patterns of responses with students can lead to better teaching and learning. Prep: Low
In class: Low
Analysis: Low
Memory matrix Students fill in cells of a two-dimensional diagram for which instructor has provided labels. For example, in a music course, labels might consist of periods (Baroque, Classical) by countries (Germany, France, Britain); students enter composers in cells to demonstrate their ability to remember and classify key concepts. Tally the numbers of correct and incorrect responses in each cell. Analyze differences both between and among the cells. Look for patterns among the incorrect responses and decide what might be the cause(s). Prep: Med
In class: Med
Analysis: Med
Directed paraphrasing Ask students to write a layman’s "translation" of something they have just learned -- geared to a specified individual or audience -- to assess their ability to comprehend and transfer concepts. Categorize student responses according to characteristics you feel are important. Analyze the responses both within and across categories, noting ways you could address student needs. Prep: Low
In class: Med
Analysis: Med
One-sentence summary Students summarize knowledge of a topic by constructing a single sentence that answers the questions "Who does what to whom, when, where, how, and why?" The purpose is to require students to select only the defining features of an idea. Evaluate the quality of each summary quickly and holistically. Note whether students have identified the essential concepts of the class topic and their interrelationships. Share your observations with your students. Prep: Low
In class: Med
Analysis: Med
Exam Evaluations Select a type of test that you are likely to give more than once or that has a significant impact on student performance. Create a few questions that evaluate the quality of the test. Add these questions to the exam or administer a separate, follow-up evaluation. Try to distinguish student comments that address the fairness of your grading from those that address the fairness of the test as an assessment instrument. Respond to the general ideas represented by student comments. Prep: Low
In class: Low
Analysis: Med
Application cards After teaching about an important theory, principle, or procedure, ask students to write down at least one real-world application for what they have just learned to determine how well they can transfer their learning. Quickly read once through the applications and categorize them according to their quality. Pick out a broad range of examples and present them to the class. Prep: Low
In class: Low
Analysis: Med
Student- generated test questions Allow students to write test questions and model answers for specified topics, in a format consistent with course exams. This will give students the opportunity to evaluate the course topics, reflect on what they understand, and what are good test items. Make a rough tally of the questions your students propose and the topics that they cover. Evaluate the questions and use the goods ones as prompts for discussion. You may also want to revise the questions and use them on the upcoming exam. Prep: Med
In class: High
Analysis: High
(may be homework)
[1] Details on these and others available from Angelo & Cross, Classroom Assessment techniques, 1993.
[2] The Bureau of Evaluative Studies and Testing (BEST) can administer the Minute Paper electronically.
Published Resources:
Angelo, T.A. & Cross, P.K. (1993). Classroom Assessment Techniques (2nd ed.). San Francisco: Jossey-Bass.
Davis, B.G. (1993). Tools for Teaching. San Francisco: Jossey-Bass.
Welcome to Classroom Assessment
Classroom assessments can include a wide range of options -- from recording anecdotal notes while observing a student to administering standardized tests. Regardless of the form, effective classroom assessments provide the feedback and motivation that students need to excel.
This course is divided into 2 parts, each containing 3 lessons. Part I focuses on basic concepts and common assessments such as multiple choice and essay questions. Part II includes performance assessment, classroom interactions, and attitude surveys. Throughout the lessons, resources and techniques for integrating technology are included.
This course can be used as a "self-study" or it can be used for Continuing Education Units (if your district has a facilitator). Each lesson contains sample assessments along with multiple activities that reinforce the concepts. Before you take the course, please fill out the Survey to help gauge your experience level. At the end of the course, a similar survey will be available.
Overview
Selected response assessment items (also referred to as objective assessments) include options such as multiple choice, matching, and true/false questions. These question types can be very effective and efficient methods for measuring students’ knowledge and reasoning. Because many of the standardized tests are based heavily on multiple choice questions, teachers should be skilled at developing effective objective assessment items. In addition, teachers should be able to construct quizzes that target higher level thinking skills (consistent with the application, analysis, and synthesis levels of Bloom’s taxonomy), and they should evaluate their instruments by conducting item analyses.
A. Formative vs. Summative Assessments
Classroom assessments can include a wide range of options -- from recording anecdotal notes while observing a student to administering standardized tests. The options can be roughly divided into two categories -- formative assessments and summative assessments.
Formative assessments are on-going assessments, reviews, and observations in a classroom . Teachers use formative assessment to improve instructional methods and student feedback throughout the teaching and learning process. For example, if a teacher observes that some students do not grasp a concept, she or he can design a review activity or use a different instructional strategy. Likewise, students can monitor their progress with periodic quizzes and performance tasks. The results of formative assessments are used to modify and validate instruction.
Summative assessments are typically used to evaluate the effectiveness of instructional programs and services at the end of an academic year or at a pre-determined time. The goal of summative assessments is to make a judgment of student competency -- after an instructional phase is complete. For example, in Florida, the FCAT is administered once a year -- it is a summative assessment to determine each student's ability at pre-determined points in time. Summative evaluations are used to determine if students have mastered specific competencies and to identify instructional areas that need additional attention.
The following table highlights some formative and summative assessments that are common in K12 schools.
Formative Assessments Summative Assessments
Anecdotal records Final exams
Quizzes and essays Statewide tests (FCAT)
Diagnostic tests National tests
Lab reports Entrance exams (SAT and ACT)
B. Writing Selected Response Assessment Items
Selected response (objective) assessment items are very efficient – once the items are created, you can assess and score a great deal of content rather quickly. Note that the term objective refers to the fact that each question has a right and wrong answer and that they can be impartially scored. In fact, the scoring can be automated if you have access to an optical scanner for scoring paper tests or a computer for computerized tests. However, the construction of these “objective” items might well include subjective input by the teacher/creator.
Before you write the assessment items, you should create a blueprint that outlines the content areas and the cognitive skills you are targeting. One way to do this is to list your instructional objectives, along with the corresponding cognitive level. For example, the following table has four different objectives and the corresponding levels of assessment (relative to Bloom's taxonomy). For each objective, five assessment items will be written, some at Level I and some at Level II. This approach helps to ensure that all objectives are covered and that several higher level thinking skills are included in the assessment.
Objective Number of Items at Level I
(Bloom's Taxonomy) Number of Items at Level II
(Blooms' Taxonomy)
1 2 3
2 3 2
3 1 4
4 4 1
After you have determined how many items you need for each level, you can begin writing the assessments. There are several forms of selected response assessments, including multiple choice, matching, and true/false. Regardless of the form you select, be sure the items are clearly worded at the appropriate reading level, do not include unintentional clues, and are not ambiguous. The validity of your test will suffer tremendously if the students can’t comprehend or read the questions! This section includes a few guidelines for constructing objective assessment items, along with examples and non-examples.
Multiple Choice
Multiple choice questions consist of a stem (question or statement) with several answer choices (distractors). For each of the following guidelines, click the buttons to view an Example or Non-Example.
• All answer choices should be plausible and homogeneous.
o Example
o Non-Example
• Answer choices should be similar in length and grammatical form.
o Example
o Non-Example
• List answer choices in logical (alphabetical or numerical) order.
o Example
o Non-Example
• Avoid using "All of the Above" options.
o Example
o Non-Example
Matching
Matching items consist of two lists of words, phrases, or images (often referred to as stems and responses). Students review the list of stems and match each with a word, phrase, or image from the list of responses. For each of the following guidelines, click the buttons to view an Example or Non-Example.
• Answer choices should be short, homogeneous and arranged in logical order.
o Example
o Non-Example
• Responses should be plausible and similar in length and grammatical form.
o Example
o Non-Example
• Include more response options than stems.
o Example
o Non-Example
• As a general rule, the stems should be longer and the responses should be shorter.
o Example
o Non-Example
True/False
True/false questions can appear to be easier to write; however, it is difficult to write effective true/false questions. Also, the reliability of T/F questions is not generally very high because of the high possibility of guessing. In most cases, T/F questions are not recommended.
• Statements should be completely true or completely false.
o Example
o Non-Example
• Use simple, easy-to-follow statements.
o Example
o Non-Example
• Avoid using negatives -- especially double negatives.
o Example
o Non-Example
• Avoid absolutes such as "always; never."
o Example
o Non-Example
B. Essay Questions (Short and Extended Response)
Essay questions are a more complex version of constructed response assessments. With essay questions, there is one general question or proposition, and the student is asked to respond in writing. This type of assessment is very powerful -- it allows the students to express themselves and demonstrate their reasoning related to a topic. Essay questions often demand the use of higher level thinking skills, such as analysis, synthesis, and evaluation.
Essay questions may appear to be easier to write than multiple choice and other question types, but writing effective essay questions requires a great deal of thought and planning. If an essay question is vague, it will be much more difficult for the students to answer and much more difficult for the instructor to score. Well-written essay questions have the following features:
• They specify how the students should respond.
o Example
o Non-Example
• They provide information about the value/weight of the question and how it will be scored.
o Example
o Non-Example
• They emphasize higher-level thinking skills.
o Example
o Non-Example
Essay questions are used both as formative assessments (in classrooms) and summative assessments (on standardized tests). There are 2 major categories of essay questions -- short response (also referred to as restricted or brief ) and extended response.
Short Response
Short response questions are more focused and constrained than extended response questions. For example, a short response might ask a student to "write an example," "list three reasons," or "compare and contrast two techniques." The short response items on the Florida assessment (FCAT) are designed to take about 5 minutes to complete and the student is allowed up to 8 lines for each answer. The short responses are scored using a 2-point scoring rubric. A complete and correct answer is worth 2 points. A partial answer is worth 1 point.
Sample Short Response Question
(10th Grade Reading)
How are the scrub jay and the mockingbird different? Support your answer with details and information from the article.
Extended Response
Extended responses can be much longer and complex then short responses, but students should be encouraged to remain focused and organized. On the FCAT, students have 14 lines for each answer to an extended response item, and they are advised to allow approximately 10-15 minutes to complete each item. The FCAT extended responses are scored using a 4-point scoring rubric. A complete and correct answer is worth 4 points. A partial answer is worth 1, 2, or 3 points.
Sample Extended Response Question
(5th Grade Science)
Robert is designing a demonstration to display at his school’s science fair. He will show how changing the position of a fulcrum on a lever changes the amount of force needed to lift an object. To do this, Robert will use a piece of wood for a lever and a block of wood to act as a fulcrum. He plans to move the fulcrum to different places on the lever to see how its placement affects the force needed to lift an object.
Part A Identify at least two other actions that would make Robert’s demonstration better.
Part B Explain why each action would improve the demonstration.
A. Fill-in-the-Blank Items
The simplest forms of constructed response questions are fill-in-the-blank or short answer questions. For example, the question may take one of the following forms:
1. Who was the 16th president of the United States?
2. The 16th president of the United States was ___________________.
These assessments are relatively easy to construct, yet they have the potential to test recall, rather than simply recognition. They also control for guessing, which can be a major factor, especially for T/F or multiple choice questions.
When creating short answer items, make sure the question is clear and there is a single, correct answer. Here are a few guidelines, along with examples and non-examples
• Ask a direct question that has a definitive answer.
o Example
o Non-Example
• If using fill-in-the blank, use only one blank per item.
o Example
o Non-Example
• If using fill-in-the blank, place the blank near the end of the sentence.
o Example
o Non-Example
Although constructed response assessments can more easily demand higher levels of thinking, they are more difficult to score. For example, scantrons (optical grade scanners) cannot score this type of assessment, and computer-based scoring is difficult because you must include all synonyms and acceptable answers. For example, all of the following might be acceptable answers to the sample question: "Who was the 16th president of the United States?" Abraham Lincoln; Abe Lincoln; Lincoln; President Lincoln; Honest Abe; the Railsplitter. You might also want to accept common mis-spellings such as Abrahem or Lencoln (depending on the objective).
Subscribe to:
Posts (Atom)