The Specific Problem with Standardized Tests

Grading a school or a teacher based on a "blind test" is the same as evaluating a football coach who cannot scout his opponents.

When the Hoosier state finally (finally) released this year’s ISTEP results, Indiana House Representative Jim Lucas (R-Seymour) wasted no time in politicizing it: “Less than half of our children can meet ‘government standards’ in a ‘government run’ education system!” he said.  A month later, when those test results had finished playing their role in the state’s “Letter Grade” calculations, school leaders struggled to rationalize and defend their respective institutions' performances.  But "defense" is the wrong strategy, and sadly it's been one that schools and teachers have been playing for too long.  It's time to go full "Rush Limbaugh" on standardized testing...not so much on the tests themselves, rather on the way state leaders such as Rep. Lucas have been using them.

Consider this example:  You’re a varsity-level football coach. The task before you is simple: Win and keep your job. This is often the analogy critics of public education use whenever teachers gripe about standardized testing.  On the surface it makes sense.  Everyone else either delivers and stays employed, or doesn’t and busts it for the unemployment line.  Why shouldn’t the classroom be any different?

But here’s the deal. You’re that football coach, and you’re preparing for the last game of the season. You have no film for scouting purposes.  You do not know about the star players. You do not know what kind of offense your opponent runs. You do not know what they typically do on first down…what they do on a third and long…what’s most likely going to happen when the tight-end motions out of the slot from the wide side on second and short.  You know nothing at all about the other team…except that they wear shoulder pads and play football.

So you practice against every hypothetical scenario you can imagine.

It gets worse, however.  On game night you’re not allowed on the field.  Confined to a far-away locker room, you don’t get to watch the game. You do not get to call timeouts. You do not get to make halftime adjustments. When the game is over the players walk through those locker room doors and vaguely tell you how they “felt” it went.  They can’t tell you any of the highlights.  Doing so would be “cheating,” so that’s forbidden.  They don’t even know the final score.

Then three months, or six months, or nine months… Sometime later you get a little bit of news (however long it takes all the football lawmakers to agree to terms and work out all the anomalies and questionable instant-replays).  At that point you finally get the score and only the score. That’s when you find out you lost the game and finished the season 5-5.  That’s when you’re told that average isn’t good enough.

This is how standardized testing works when you use it to evaluate schools and teachers.

The problem isn’t that standardized tests don’t do what they are meant to do.  If the only goal of the imaginary football scenario described above is to measure the raw, athletic ability of the players on the field (and perhaps measure their cognitive ability in terms of naturally reading and reacting to opposing formations), then a “pick-up” style such as that would work perfectly.

But we don’t use that season for its intended purpose.  We abuse and “square-peg/round-hole” the hell out of it.  And because it seems simple, we don’t argue.

And then we compare those test scores to the students’ grades (of course we do).  And we ask the obvious question:  If these kids are doing so badly on the big tests, why are they getting good grades?

There’s answer for that, too, as one of my favorite college classes illustrates:

Every third Friday I sat in Dr. Charles Rinehart’s Survey of British Authors class and frantically scribbled five explication essays in a wide ruled blue book. Rinehart—a wizened, rotund, and likable fellow—had prepared us well for each of those tests.  Over the course of each three-week unit we read, studied, and discussed some 20-25 poems, stories, and excerpts.  Rinehart had culled them from a pair of 1,200-page anthologies printed on paper so onion-skinned that you could see your fingerprints when you put your hand up to the other side. Of those two-dozen works, Rinehart picked 10 passages for the test. Of those 10 we picked our “blue book five.”

Our job: spot the passage, name the work and author, discuss all relevant themes and literary devices, and connect the work to the world in which it written.  We practiced the art before the first test. We debriefed after every exam as well.  Rinehart was a master of his universe. When he taught us he was exact and deliberate in his preparation.  Every theme.  Every metaphor.  Every conceit. When we handed him our blue books he saw the results of our work immediately, and he was able to break down what went well and what didn’t in explicit detail before the next test.  Furthermore, we were all English majors both well versed and experienced in the art of literary analysis. This was our milieu.  Yet we still needed long, in-depth discussions to work our heads through poems we studied.

Standardized testing, as previously established, works nothing like this.

As a case in point (and sticking with poetry as our subject matter), consider one example: the AP literature and composition exam. In that test, high school students study poetic forms and devices in a general sense. They do not cover specific poems for the test, and they barely (if at all) cover the relationship between specific poems and the historical world view in which they were written. Teachers preparing students for this exam have absolutely no idea what the College Board is going to put on the test. Of the literally hundreds of thousands of available poems the College Board is going to pick three.

The coach is preparing for the team he cannot scout, for the game he won’t get to watch.

[divider style="solid" top="20" bottom="20"]

For all the talk about “accountability” what we have really been doing all this time amounts to little more than a glorified game of Pin the Tail on the Donkey.

And returning to that woeful position as Hypothetical University’s football coach let’s add a few more caveats:  You can’t pick your best player as quarterback.  Every player has to start.  Every player is expected to run, catch, and throw touchdowns.  Every player is expected to drill a 60-yard place-kick through the uprights.  If players skip practice, they still play, and their performance is held against you.  If players half-ass their way through the game (not that you’ll know since you’re not there), you still have to play them again.

When people dismiss criticisms of standardized testing they do so because they’re looking at standardized testing in idealized vacuums. No lawyer would walk into a courtroom without knowing every answer to every question before the first syllable is uttered.  No surgeon will start that triple-bypass operation without knowing as many specific details as possible.  Contractors don’t show up to the job site without plans and wing it.  NASA didn’t shoot people to the moon based on guesswork.

Whining and arguing about standardized testing is no longer good enough.  To the general public, that comes off as people who are pouting because they didn’t get the results they wanted.  It sounds like people who are running away from responsibility, unwilling to be held accountable for their work.  The debate has to shift.  It’s time to attack the abuse and misuse of standardized tests, and it’s further time to question the reasoning of school systems who validate them.  As long teachers, administrators, parents, and community members put on the blindfold and play the game, the state will keep mandating more tests—always “revamping” them every time schools have figured out how to game them.

For all the talk about “accountability” what we have really been doing all this time amounts to little more than a glorified game of Pin the Tail on the Donkey.  To see it any other way is naïve.  To stick by it when you should know better is something much worse.

[Wheeler proudly teaches AP Language to some bright and lovably obnoxious kids in a small college town. He also contributes to the craft beer website Indiana on Tap and writes for other publications. He started learning to play guitar last fall, but he remains terrible at it. 

Featured Image: Test-986769 640 by lecroitg is a Public Domain image.