Pharyngula::Bugger. I had dang near perfect SAT scores, too.

Pharyngula has moved to http://scienceblogs.com/pharyngula/

Wednesday, May 04, 2005

Bugger. I had dang near perfect SAT scores, too.

PZ Myers • 65 Comments (last page)

All right. It's over. I give up on America. We're too stupid to live.

Perelman studied every graded sample SAT essay that the College Board made public. He looked at the 15 samples in the ScoreWrite book that the College Board distributed to schools nationwide to prepare students for the essay. He reviewed the 23 graded essays on the College Board website meant as a guide for students and the 16 writing "anchor" samples the College Board used to train graders to properly mark essays.

He was stunned by how complete the correlation was between length and score. "I have never found a quantifiable predictor in 25 years of grading that was anywhere near as strong as this one," he said. The shortest essays, typically 100 words, got the lowest grade of one. The longest, about 400 words, got the top grade of six. In between, there was virtually a direct match between length and grade.

He found that he could also grade the essays with a good correlation with their awarded scores just by looking at their shape and length—he didn't have to actually read them.

It gets worse. Take a look at the official policy of the College Board.

He was also struck by all the factual errors in even the top essays. An essay on the Civil War, given a perfect six, describes the nation being changed forever by the "firing of two shots at Fort Sumter in late 1862." (Actually, it was in early 1861, and, according to "Battle Cry of Freedom" by James M. McPherson, it was "33 hours of bombardment by 4,000 shot and shells.")

Perelman contacted the College Board and was surprised to learn that on the new SAT essay, students are not penalized for incorrect facts. The official guide for scorers explains: "Writers may make errors in facts or information that do not affect the quality of their essays. For example, a writer may state "The American Revolution began in 1842" or " 'Anna Karenina,' a play by the French author Joseph Conrad, was a very upbeat literary work." (Actually, that's 1775; a novel by the Russian Leo Tolstoy, and poor Anna hurls herself under a train.) No matter. "You are scoring the writing, and not the correctness of facts."

How to prepare for such an essay? "I would advise writing as long as possible," Perelman said, "and include lots of facts, even if they're made up." This, of course, is not what he teaches his MIT students.

Students are not penalized for incorrect facts.

I felt like clawing my eyes out when I read that, but fortunately the gusher of tears made it impossible to get any traction.

Remind me to ignore SAT scores forevermore.

Trackback url: http://pharyngula.org/index/trackback/2246/QBcPKJZW/

Comments:

#23977: Robn — 05/04 at 11:27 PM
I trained a few years ago to grade the writing scores of SATs because, well, I was tired of driving 2 hours each way and getting paid $4k a year as an adjunct professor.

Needless to say, as I trained, I realized that I was grading essays entirely incorrectly: I wanted them to be good. ETS had a ridiculously strict grading system, based, as you say, on style over content.

I couldn't justify rewarding the next generation of postmodernists, so I never took the final job of grading. I may have had no money, but at least I could sleep at night!

#23978: — 05/05 at 12:03 AM
ArtK:"One of my big mistakes was skipping some calculus by virtue of my AP scores -- I was unprepared for the more advanced courses and university life and taking basic calculus "over" would have helped with the transition."

I took calculus as a HS senior, using the text with the chambered nautilus on the cover (surely someone knows the one). When I started college, I took Cal 1 and 2 out of the same book, but this time I "got it." When I got to Advanced Cal 457 (Magnum!), I downright enjoyed it! Of course, now I can't remember it.

#23979: — 05/05 at 12:17 AM
I disagree strongly with PZ and others about the facts question. The ability to recall facts is not germane to writing ability. The essay portion of the SAT is intended to measure writing ability. Correctness is therefore not an acceptable criterion for grading.

These are not serious essays. They're a way of demonstrating that you, unlike the vast bulk of American high school graduates, can formulate an intelligible sentence.[1] That is all.

D'A

[1]: In theory. In practice, it seems clear that they're really testing your stamina.

#23980: — 05/05 at 12:57 AM
Good writing makes reading joyful. Good writing gets the point across -- it does, indeed, have a point to it. But because it has a point that people understand, good writing tends to make some people angry.

This is good if one is a speechwriter, or press release writer, and one's goal is to stir others to action.

In a few places, good writing is punished. I had a supervisor complain that my writing was too "colorful," and had earned the attention of her supervisor, who kept asking questions about the topics I wrote about, and not the topics my supervisor wanted to talk about. I volunteered to color up her favorite stuff, but she missed the point.

While earning my crusts turning government and legislative drivel into news stories, my first-year legal writing instructor had fits. She expected a full page of fog, and I delivered three short, crisp paragraphs. Worse, she told me on the paper, I had changed her mind about that point of the law the brief had covered. It was one of the best Ds I ever got.

So, of course there is a point to writing in school. The difference, I suppose, is whether one is writing to educate one's self, or simply to get a few points and get through the class. It appears the SAT folk have chosen the latter path. Scrap the damned exam.

#23983: mattH — 05/05 at 02:50 AM
In practice, it seems clear that they're really testing your stamina.

Speed as well. I'm sure that it's timed, so writing quickly is going to be an asset too. BS'ers paradise.

#23984: Republic of Palau — 05/05 at 03:10 AM
This reminds me of my one and only foray into academic skullduggery: I wrote my sister's dissertation on the role of NGO's( she had serious family issues right before deadline, and it was based on her research), for her degree in development studies, despite it being a subject I had no knowldge of. It got a 2:2, which wasn't bad at all, considering.

#23986: Alon Levy — 05/05 at 04:10 AM
The article is less than worthless, because it doesn't say how well length and grade correlate, and how well they should ideally. Gut feelings about correlation don't qualify as serious argument; exact correlation coefficients do.

#23989: — 05/05 at 05:01 AM
I remember something about a guy who, asked to write an essay about the French Revolution, talked about its tangential relationship with the history of some race of imaginary aliens he made up. He got a perfect score.

#23990: — 05/05 at 05:07 AM

And what i find even worse is that I was thinking of reading "Anna Karenina". Now I might not bother since I know the ending.

If I recall correctly, the suicide is not the ending, but the beginning. It's still worth a read.

#23991: Alon Levy — 05/05 at 05:57 AM
Is there any chance you can hunt the reference down, Julian?

#23992: — 05/05 at 06:46 AM
"The purpose of writing is to inflate weak ideas, obscure pure reasoning, and inhibit clarity. With a little practice, writing can be an intimidating and impenetrable fog!"

- Calvin, from Calvin & Hobbes

#23993: — 05/05 at 07:03 AM
Don't claw your eyes out Mr Myers! Just keep your eyes open when you sneeze and they'll pop out by themselves.

#23996: ajmilne — 05/05 at 08:21 AM
I started to respond to this by writing a really long post... with extremely long sentences (yes, longer than my usual ones), all based on premises I'd pulled entirely out of my lower digestive tract. And it was a fascinating world I found myself working from--in it, Hitler was actually Swedish, and Yuri Andropov was the drummer for the Jimi Hendrix Experience.

But it didn't come out particularly funny, so I'm sparing everyone.

(Besides, I thought maybe I should save it for later use, in the unlikely event I wind up writing the SATs someday.)

#23998: PZ Myers — 05/05 at 08:38 AM
Several people have made the entirely reasonable assertion that the SATs aren't testing mastery of the facts, but simply writing ability. I have to disagree. Writing isn't an isolatable skill floating off independent of all other abilities, and we shouldn't be treating it as an exercise in linking words together. Writing is about expressing ideas. Without some structure and supporting argument, it becomes the free-form, dissociated babble of schizophrenics. That is not what I want to see encouraged.

I'm not speaking as a science chauvinist, either. A love poem also has to touch on reality (a nice, idealized, airbrushed reality, perhaps) now and then in order for the recipient to recognize it.

It's a real problem, too. I get papers all the time where it's clear that the student has some grasp of grammar and spelling, but doesn't have a clue about how to organize his thoughts and build a rational story. And that's what we all want from our writers: the ability to construct a comprehensible narrative or mood or argument, not the ability to fill a Scrabble board with letters.

PZ Myers
Division of Science and Math
University of Minnesota, Morris

#24000: — 05/05 at 08:43 AM
On facts: The SAT shouldn't be testing for factual knowledge. I consider it to be an error in writing when one asserts a fact that one does not know. So errors in fact don't just reflect a lack of knowledge; they reflect a lack of knowledge of the principle that you shouldn't assert knowledge when you really have none. How can we tell if test-takers are simply making up facts or made a simple mistake? I don't know. But there must be some way to distinguish between the two, and we should penalize people for making things up. Its dishonest, and contrary to the principles of academic integrity that universities aspire to.

#24002: — 05/05 at 09:00 AM
<<He was also struck by all the factual errors in even the top essays. An essay on the Civil War, given a perfect six, describes the nation being changed forever by the "firing of two shots at Fort Sumter in late 1862." (Actually, it was in early 1861, and, according to "Battle Cry of Freedom" by James M. McPherson, it was "33 hours of bombardment by 4,000 shot and shells.")>>

Jeebus X. Kristmas - the kid has obviously never seen a photo of the fort after the bombardment. Sumter was almost flattened; the Union soldiers crawled out from rubble. I think the most interesting thing about the start of "the late unpleasantness" was that the only death after 33 hours of bombardment was one horse. As David McCullough stated in the beginning narrative to Ken Burns' Civil War documentary "It was an oddly bloodless beginning to America's bloodiest war."

{I am not discounting the life of the horse - so don't send PETA after me}

#24006: Roman — 05/05 at 09:21 AM
Wow. Just wow. I'm really glad right now that I'm taking the International Baccalaureate. *shudder*

#24010: Dr Pretorius — 05/05 at 09:25 AM
I'm not really the slightest bit surprised by that, and I suspect the same holds for the GREs. I've always (well, twice) scored perfect on the writing section for the GREs, but I can't really think that has much to do with ability. There's some minimal sense in which it does, but frankly I learned early on that when you're writing an essay for someone who's trying to assess your ability and not you knowledge, and someone who doesn't really care what you said the trick is to use big words and lots of complicated sentences.

The first sentence I wrote on my latest GRE writing section was probably longer than the above paragraph, and the last thing I did before submitting it was go back and change all the simple, clear words that expressed my ideas with bigger, fancier ones that didn't do it as well.

Standardised testing is, as far as I can tell, in large part knowing how to game the system.

In reference to the earlier porcupine story, I should add that as someone who does grade humanities papers it wouldn't surprise me if one of my students was trying that sort of thing. The fact to know is that, as a grader, I really don't care. I might mark it down if I'm in a marking mood, but usually I'll just roll my eyes and move right along to trying to assess the argument itself. It's more than possible that a lot of your professors did the same thing...

#24011: Alon Levy — 05/05 at 09:34 AM
Roman, IB is never compared with the SAT, and in US universities is not an SAT substitute but a high school GPA substitute.

#24014: — 05/05 at 10:08 AM
>the nation being changed
>forever by the "firing of
>two shots at Fort Sumter
>in late 1862."

If you want essays to be fact checked, I think there are a few practical problems to be solved. The civil war quote is a good example:

If you marked me down for “Two Shots”, I would reply: "Of COURSE I know there were thousands of shots fired. But I was talking about the FIRST shot fired by the south, and the FIRST shot in reply from the fort. With THOSE those two shots the nation was at war, and forever changed!"

What about an unambiguous error, like “in late 1862”?.. A teacher in a history class should know her subject and recognize mistakes. But is knowledge of the Civil War a job requirement of the SAT graders? Are they also required to know about Biology? Astronomy? English Literature? Philosophy? Everything else that any test-taker might choose to write about?

When they don’t know the subject, how much time do they have to research each paper? If they aren't sure about a fact, do they mark down for it or let it pass? If they let it pass, does this give an advantage to students who write about really obscure subjects? Do they mark down if the subject is a hotly disputed one, and the student takes the other side from theirs? If the student really believes some silly statement (for religious reasons, say) could you be facing lawsuits if you mark it down?

#24019: — 05/05 at 10:40 AM
>>If you want essays to be fact checked, I think there are a few practical problems to be solved. The civil war quote is a good example:<<

I don't think the essays should be fact-checked. I think they should be "bullshit" checked. I don't think test-takers should be marked down for a factual error. But "just making it up" should be penalized.

#24020: QrazyQat — 05/05 at 10:44 AM
I once heard a fellow (it may have been Jerry Pournelle) talking about a study he had been involved in checking on how folks had done in some organization he'd been in and the one thing that correlated most accurately was the length of their resumes or vitae. Not quality, or importance of the work done, just length.

And when I wrote a computer book 20 years back, as I shopped it around, not one editor actually looked at it, even those who were enthused, even the ones who bought it (and were computer experts themselves). The one thing every last editor did do was to heft the manuscript and weigh it in their hand. Every last one.

#24029: — 05/05 at 11:24 AM
One of the banes of my existence is that my introductory classes can also be taken at the 300-level to satisfy an "intensive writing" requirement. I'm not allowed to make them do too much more work, I'm just supposed to help them with their writing.

As it happens, I have a secret identity as an editor, so I'm in a better position than most, but we assume an anthropologist is qualified to teach writing, I don't know.

I have them do 5 assignments based on short, paired articles related to topics we're covering in the course that week.I mark them extensively for grammar, punctuation, usage, clarity, and organization. I point out how they could reorganize to make points more persuasive, I urge them to engage in an integrated discussion of the two articles and, for the love of Baal, don't just serially describe them to me.

I don't put letter or number grades on them. So long as they show improvement and attention to comments, they get their "S" for that part of the class.

Best comment ever on this torture: "She grades papers REALLY HARSHLY like this is a Master's-level course!"

#24039: craig — 05/05 at 12:08 PM

The article is less than worthless, because it doesn't say how well length and grade correlate, and how well they should ideally.

But the sentence structure was nice, so we're going to consider it a good article.

#24061: CC — 05/05 at 02:30 PM
(Actually, that's 1775; a novel by the Russian Leo Tolstoy, and poor Anna hurls herself under a train.)

Oh, yeah, spoil the ending for me. Geez, some people.

Page 2 of 3 pages < 1 2 3 >

Next entry: Wells and Dembski: two phonies

Previous entry: What makes humanity special: balls and lymph nodes

<< Back to main

Pharyngula has moved to http://scienceblogs.com/pharyngula/

Wednesday, May 04, 2005

Bugger. I had dang near perfect SAT scores, too.

Info