Teacher and coach

Math teacher does item analysis, finds it useful

About halfway thru last semester I discovered that there’s a relatively simple way to perform an item analysis on the mid-term and final exams in my classes.  An item analysis basically means examining student responses to each of your test questions so that you can improve the quality of your tests.  The most basic type of item analysis involves simply seeing what percentage of your students missed each one of your exam questions, thus allowing you to determine if some of your exam questions are too difficult, or if there are certain topics that maybe your students didn’t master as well as you had expected.

It turns out that for the Scantron machine that my school uses, after you scan all the student test forms you can just scan a special Item Analysis Form and the machine will give you a nice printout that shows how many students missed each question.  (Obviously, this only works for multiple-choice tests, and only if you’re grading them using the Scantron machine.)  I’m not generally a big fan of multiple-choice tests so all of my other tests are hand-graded, but my mid-term and final are both multiple-choice.

After scanning the item analysis form for each set of final exams, I wrote down all the questions which were missed by over half of the students.  A couple of things immediately jumped out at me:

1. The questions which I had answered incorrectly on my exam key (oops).
2. The questions which I thought were really easy, yet which over half of the class missed.

I thought I had a pretty good idea of which exam questions were easy, medium, and hard, but after looking at the results of the item analysis I had to re-evaluate.  There were at least a couple of questions on the exams for each of my classes that even now have me scratching my head.  How could so many kids have missed those questions?  I was sure everybody was clear on those topics.

As disappointing as it was, however, it was a very valuable experience.  I may have thought those problems were pretty straightforward, but since most of the class missed them on the final I’m going to have to re-work some of the lessons dealing with those topics.  I would also like to do a similar item analysis of each of the other tests in my classes, though since they’re not multiple choice I’ll have to do that without the Scantron machine.

[BTW, the book Statistics Hacks by Bruce Frey has some good info on item analysis and other related statistical stuff that teachers can use to improve their tests.  I got a used copy for 5 bucks on Amazon.]

Image by COCOEN on Flickr.

4 Responses to “Math teacher does item analysis, finds it useful”

  1. bledsoe says:

    I would LOVE to have all my students acing all my tests! Though “if I have the time” is of course the real sticking point. Still, it’s certainly something to shoot for, isn’t it?

  2. Allen says:

    If you have the time, I’d suggest analyzing these tests every single time. It will make you a better teacher in no time. Only by analyzing these tests do you know what the biggest misunderstandings are, so you can fix them next year. At some point everyone in your class will ace all your tests ;D

  3. bledsoe says:

    In fact, I already have in mind to do something pretty similar to “sitting them down and having them talk thru those questions,” just by paying more attention to those topics in my lessons next semester, having students walk thru those problems more slowly, not assuming they understand something that seems obvious to me, etc.

    I suppose every time teachers give tests they’re conducting a kind of “user study,” though not nearly as in-depth as what you describe; the 1:30 teacher:student ratio, plus the limited time resources that teachers have to deal with, makes that prohibitive. But we still take notes and then scurry off to try to make things better.

  4. browse says:

    In a world of infinite time, I’d love to sit down a few students and ask them to do some of those “surprising” questions in front of you, talking out loud as they do it. It would be fascinating to hear what’s going through their mind as they attempt to pick the problem apart.

    In the software biz, we would do something similar, called “user studies”. We’d spend months and months finely crafting a user interface for some piece of software, and then drag in poor innocent users to watch them attempt to complete some tasks using the new software. There’s a very careful protocol, where you do your damnedest to avoid coaching them through how to use the product; all you do is give them an end goal to accomplish and then sit there and cringe while they completely fail to do even the simplest task with your shiny new user interface. And you bang your head on the desk as you watch. And you cry little. Then you take copious notes and scurry off to try to make things better. I always found it to be a fascinating process.

Leave a Reply

Your email address will not be published. Required fields are marked *

Powered by WordPress | Designed by Elegant Themes