What are your best tips for writing tests?

Multiple-choice questions

• From Margaret Weck
• Part of the discussion about multiple-choice test construction and whether or not to use multiple correct answers arises from the requirements for determining the validity and reliability of the tests we are constructing. It is difficult to statistically verify the validity of questions (that question 2. actually tests student’s mastery of concept Y) when you have five interlocking elements that are required to correctly answer the question. A correct answer may mean the student has all the required elements mastered, or it may mean she is a good guesser. An incorrect answer does not help the instructor pin-point WHICH of the five elements is missing or weak or inaccurately applied. If the purpose of testing is to simply “sort” students by relative mastery, then it doesn’t matter as much. However, if the purpose of the exam is to document mastery of specific pieces or content or application of particular theories, then the less ambiguously the questions are phrased, the better. The amazing thing I have found is that student performance does still vary according to ability even with “straight-forward” questions. It is very difficult and time-consuming to get well-crafted multiple-choice questions that appropriately test for application of principles or physiological mechanisms.
• Thoughts from Kevin (from a message to the HAPS-L listserv)
  
• In analyzing how my students respond to questions I have found that adding a "none of the above" statement as a last choice makes significantly more people miss that question, so much so that I have stopped using that as an option (though I continue to have "all of the above" available for some questions). The other thing I carefully avoid now are questions that say, in one way or another, "All of the following are true EXCEPT," or "All of the following are false EXCEPT." Those are very simple questions to write but inevitably a lot of students make dumb mistakes on them. My rule of thumb is to write a question that a student can formulate an answer to BEFORE reading the various options. In looking through my test questions, I realize that most of them are still built around recall of facts and not about applying information in a new context. Here are a couple example questions that are closer to my personal "ideal" multiple choice question:
  
  1. You arrive at the scene of a car accident on a summer afternoon. You see that the driver is conscious but bent over in pain. You then notice paradoxical motion in the thoracic region upon inhalation. You conclude:
  a. the head of a rib has dislocated from the fovea on the body of a thoracic vertebrae
  b. the odontoid process has been fractured
  c. the driver has a compression fracture in the thoracic vertebrae
  d. the driver has a LeFort fracture
  *e. the driver has broken at least 3 ribs
  
  2. “Let’s go NOW!” says the tired mother as she yanks on her son’s arm. Later her son complains to her about pain near his elbow and she notes swelling on the lateral side, just distal to the humerus. What has most likely happened?
  a. luxation of the ulna from the trochlear notch
  b. swelling of the ulnar nerve
  *c. head of the radius has popped out of the anular ligament
  d. a fracture along the epiphyseal growth plate at the distal end of the humerus
  e. dislocated shoulder joint has caused pressure on a nerve which is felt in the elbow
  
  
  One last note: I am a very good guesser on multiple choice exams, and there are natural patterns in questions that one can look for in order to guess correctly. Being aware of these patterns, I try to avoid them in my own questions. Here are some things I consciously look for when guessing on a multiple-choice test:
  -If one answer is longer than the others it is more likely to be correct (you will notice in my questions that I am careful to avoid this common pattern)
  -If two answers are very similar or are saying the same thing in some way, it is likely neither of them
  -If two answers are opposites, it is likely one of those (this is surprisingly common)
  -If "all of the above" or "none of the above" is a choice it is less likely to be the answer (usually just a distractor) unless it only shows up rarely on the test
  -There are usually some key words within the question, which may give the answer away
  
  Obviously by examining the question and responses carefully a person can do better at guessing than just by looking for those patterns. But my point is that test taking is a separate skill than having an understanding of the material. I frequently outperformed my peers on exams even when they clearly had a better understanding of the material, simply because whenever I took a test I thought of it as a fun game, a puzzle to play with, while they were suffering from fear-induced brain lock (I did study too, but these guessing skills made me look much smarter than reality). Seeing this disparity between test scores and understanding, I now have great agony when I write tests. I am not so concerned about students who may not know the material being able to guess the right answers (very few of them are good testers) as I am concerned about the students who have really worked to understand the material but simply have lousy testing skills. When the low scores come rolling in I am at a loss as to how much is a result of my teaching, how much is a result of my test writing, how much is a result of the students' studying and understanding, and how much is a result of the students' testing skills.
  

Short-answer questions

Essay questions