About halfway thru last semester I discovered that there's a relatively simple way to perform an item analysis on the mid-term and final exams in my classes. An item analysis basically means examining student responses to each of your test questions so that you can improve the quality of your tests. The most basic type of item analysis involves simply seeing what percentage of your students missed each one of your exam questions, thus allowing you to determine if some of your exam questions are too difficult, or if there are certain topics that maybe your students didn't master as well as you had expected.
It turns out that for the Scantron machine that my school uses, after you scan all the student test forms you can just scan a special Item Analysis Form and the machine will give you a nice printout that shows how many students missed each question. (Obviously, this only works for multiple-choice tests, and only if you're grading them using the Scantron machine.) I'm not generally a big fan of multiple-choice tests so all of my other tests are hand-graded, but my mid-term and final are both multiple-choice.
After scanning the item analysis form for each set of final exams, I wrote down all the questions which were missed by over half of the students. A couple of things immediately jumped out at me:
1. The questions which I had answered incorrectly on my exam key (oops).
2. The questions which I thought were really easy, yet which over half of the class missed.
I thought I had a pretty good idea of which exam questions were easy, medium, and hard, but after looking at the results of the item analysis I had to re-evaluate. There were at least a couple of questions on the exams for each of my classes that even now have me scratching my head. How could so many kids have missed those questions? I was sure everybody was clear on those topics.
As disappointing as it was, however, it was a very valuable experience. I may have thought those problems were pretty straightforward, but since most of the class missed them on the final I'm going to have to re-work some of the lessons dealing with those topics. I would also like to do a similar item analysis of each of the other tests in my classes, though since they're not multiple choice I'll have to do that without the Scantron machine.
[BTW, the book Statistics Hacks by Bruce Frey has some good info on item analysis and other related statistical stuff that teachers can use to improve their tests. I got a used copy for 5 bucks on Amazon.]
Image by COCOEN on Flickr.