#23942: — 05/04  at  03:50 PM
Is there a corporate link between the ETS at the College Board and the DI? it sure sounds like the former taught the latter how to write—especially that facts don't matter. Make up any you need to support your argument.

#23943: — 05/04  at  03:56 PM
I thought SAT scores werent worth very much for the same reasons that IQ tests arent.

And what i find even worse is that I was thinking of reading "Anna Karenina". Now I might not bother since I know the ending.

#23944: — 05/04  at  03:58 PM
I was once enlisted to score state high school Mathematics tests. After sitting through a one day seminar on scoring in which it was apparant that lots of things were considered more relevant than actual mathematical ability, I resigned. A room full of math teachers could not agree on many of the answers, partially because we couldnt agree on what the questions were actually asking.

Thanks for reminding me of this, I guess.

#23945: Socar — 05/04  at  04:32 PM
Two little observations, here:

1) In what world does 100 words constitute an essay? Man, that's hardly a decent-sized PARAGRAPH! I'll probably use more than that just writing this comment. (Then again, I can be a little on the windbaggy side. "Windbaggy" shouldn't count towards my total, though, since it isn't technically a word.)

2) This is absolutely true. Every essay and exam I ever turned in over the course of my higher academic career (about 7 years' worth, all told), contained the word "porcupine." These were not, one might note, essays ABOUT porcupines. I'd just stick it in randomly. "This type of painting is often described as 'frontier porcupine'. Note the sweeping vistas, and forbidding precipices. The artist...[etc]." Once--that's right, ONE TIME--my essay came back with the word circled in red, and a question mark next to it. That was the only time it was ever noted.

Further evidence of professorial non-perusal of essays on boring topics: I once got an A on an art history paper which I later discovered was factually incorrect in every possible way. It even placed the artist under discussion in the wrong century, and associated him with a movement that would not begin for another forty years. The professor made no note of this glaring mistake.

Same thing applies in maths: answers are generally checked, but algorithms may not be. I tested this by adding garbage variables to various equations, and eliminating them by the final line. They were circled more frequently than the porcupines, and occasionally resulted in marks being docked, but they often slipped through unremarked.

Also, in high school, I handed the same English paper in to the same teacher two years running. She didn't notice.

My theory is this: in humanities classes, lazy professors check for a reasonable proficiency with the English language, and an apparent grasp of the subject matter during classes. If these conditions are present, they do not read assignments. They may skim, in search of egregious flaws, but sometimes even this step is skipped. In maths and science, the opportunities for laziness are not as obvious, but some professors may take the easy way out by checking only final answers (not the reasoning behind them).

Finally, an IQ-test anecdote: I had one done at a very young age. For some reason, during the test, I was asked to define "gambling". Why they'd ask a little kid this question, I do not know. My answer was "getting drunk in a pub". Go figure. I knew it was SOMETHING bad that adults did, but I was hazy on the details, there.

#23946: Socar — 05/04  at  04:35 PM
PS - I may have told you the porcupine story before. I'm really lame that way. I always tell the same stupid stories over and over again.

#23947: bigo — 05/04  at  04:41 PM
i dunno, frankly i think this is pretty reasonable (the part about incorrect facts, not about length<->grade correlation). I assume these essays are being administered in the usual SAT fashion, which is to say that the whole thing has to be made up on the spot. The point of the test *really is* to judge a student's writing ability, not to determine if they know a certain piece of information. In fact, that's the whole point of the SAT - it judges students' abilities to reason and deduce, not to recall facts.

Besides, SAT questions need to have relatively straightforward answers, since you can't really allow for differences among graders. I mean what if some kid wrote his essay on Intelligent Design and got a bio teacher as a grader, and content was actually an issue? Of course this doesn't excuse the apparent grade<->length correlation, but the gist remains the same - you have a GPA which is largely based on tests which are largely based on factual recall, and you have an SAT score which is largely based on deductive reasoning / basic mathematical skills.

#23949: ArtK — 05/04  at  05:01 PM
I agree that the correlation between length and grade is horrifying, I don't have the same reaction to the facts issue. This part of the SAT isn't about facts, it's about writing clearly. Wouldn't the ID-trolls that you get be more entertaining if they could actually write, even if their facts were dead wrong?

I'm bothered a bit by the bad facts in the exemplars, though. It's one thing to tolerate poor facts when you're grading writing, but another to distribute examples that are factually wrong.

#23950: ArtK — 05/04  at  05:12 PM
The whole standardized-testing thing is a crock in any case. Anecdote: I got excellent SAT scores but was only mediocre in college. Why? Because I'm a good test taker and a mediocre student.

My kid's school (private, secular) is dropping AP classes in 2007/2008 because the tests really aren't that valuable. The term they used was "an inch deep and a mile wide," which means that classes teaching to these tests can't do anything in depth -- you have to spend your time memorizing junk to get the breadth necessary. A lot of universities are giving less credit for good scores on the AP tests than they used to. One of my big mistakes was skipping some calculus by virtue of my AP scores -- I was unprepared for the more advanced courses and university life and taking basic calculus "over" would have helped with the transition.

#23951: — 05/04  at  05:30 PM
I'm so lucky! My geology professors read (and mark up) every word. "A" papers come back dripping with ink, marked as one would edit a colleague's draft of a submission to a journal. Lesser papers have every margin inch scribbled. I've had several professors do this, even in undergraduate classes (though they may slack (?) off a bit in their lower-division general education classes, where class sizes are larger and students are known to NOT read the comments).

#23952: Virge — 05/04  at  05:34 PM
Anyone like the idea of behaviorism?

School training:
Year 1: Make up facts + write a lot = reward
Year 2: Make up facts + write a lot = reward
...

After graduation:
Hmmm. I wonder what I'll do to earn a crust...

#23953: — 05/04  at  05:34 PM
wait, i disagree about SATs being a crock. If you are good at the SATs, you are good at what the SATs test for. They aren't meant to prove that you are a good student; mainly only your grades can do that. I had a 2.6GPA in high school (that's not good), but with a 1540 on my SATs I was at least able to go to a university. And true to form I wasn't a very good student until recently; but the SAT was never a guarantor of my willingness, only my aptitude.

Trackback: Standadized testing has met it's SATuration point Tracked on: all about athenus.com (69.196.244.16) at 2005 05 04 18:09:08
Well, it's often been said that standardized exams do little but test how familiar people are with the particulars of that given exam. Entire careers are based around familiarizing people with testing techniques and making them good at guessing whic...

#23956: — 05/04  at  06:11 PM
I think I have to side with bigo here. I predate SATs in my country, so I don't really know what they're supposed to be for, but if the students are given a free rein to write about any subject, then not only are the absolute facts irrelevent, but fact-checking them is going to be a nightmare.

Honestly, PZ, if you were marking these essays, would you be able to hold a student who writes about evolution to the same standard as a student who writes about medieval french poetry, say, or graphic design, or whatever it is the student feels confident enough to write an essay about?

I don't know anything about the US education system, but surely students are still being tested factually on the subjects they're taught as well?

Oh, and the password beginning "chromonem-" doesn't fit in the little window....

#23958: — 05/04  at  06:40 PM
It's not just the SAT. I took the new GRE last year (it has a writing section), and wrote a small essay on some or the other meteorite impact in the early 1000s in China. The overall theme of the essay was to discuss whether it might have actually been a volcanic eruption instead of a meteorite. So I proceeded to outline the differences in resultant geology from the two, and concluded that geologists should go out and just look for themselves rather than basing their findings on old Chinese texts.

The GRE board was not impressed. It seems that the scientific method doesn't carry as much weight with the graders of those essays as a science student might assume. They apparently favor windbagginess over simplicity.

#23959: — 05/04  at  07:15 PM
Dang! Way long time ago I got 800 on the SAT English achievement test (the one you take in the afternoon, which included an essay).

It was a curse. From then on, every teacher expected me to be GOOD.

And now you're telling me I was just windy?

If I'd written fewer paragraphs, I could have slid through Cow College in my mediocre fashion and nobody would have expected anything better out of me?

Dang and double dang!

#23960: decrepitoldfool — 05/04  at  07:16 PM
I'm sorry, I just can't ignore the facts thing. I don't care how good your writing ability is - if you write down wrong facts you should be downgraded for it. Facts matter.

I could flex as far as allowing the writer to say, "Tolstoy published Anna Karinina in 18xx" with the understanding that it would be looked up later but using wrong facts as placeholders for facts to be looked up later invites corruption of the information stream.

#23961: decrepitoldfool — 05/04  at  07:18 PM
Hmm, not as clear as I'd hoped... you can downgrade my reply. I meant to say; placeholders are OK if they are very obvious. Non-obvious placeholders should be downgraded.

#23962: — 05/04  at  07:44 PM
Length and sentence length is often thought of as typical of intelligent, informed and "good" writing. I spent much of my professional life as a textbook editor in which capacity I corresponded with many K-12 teachers and administrators, and members of university education faculties. Some were good writers but many saw good writing as what I described in my opening sentence. The longer the sentence, the more convoluted the construction, and the bigger the words, the better they wrote—in their eyes. Was it hard to convince them otherwise.

#23966: Jeff — 05/04  at  08:30 PM
I'm torn. I do agree that it's preposterous that you can write a bunch of utterly incorrect facts in an essay that - in part - can determine what university you can get into, and get a good score.

On the other hand, I do agree that the SATs are not about "knowing facts."

I actually taught students at the Kaplan Test Preparation Centers for several years (both in class and as a one-on-one tutor), and it certainly opens your eyes to these standardized tests. I taught test-taking for the SATs, the SAT IIs, and the ACTs - in addition, I sat through an LSAT course as part of my training.

The truth is far worse than some on this thread have mentioned - the SATs are not really testing "deductive reasoning" and such. The only thing the SATs really test is how good you are at taking the SATs. That's why outfits like Kaplan and the Princeton Review can offer you tips and gimmicks to get a good score. Sure, some innate capacity for abstract reasoning helps, but if you've got the gimmicks, you can get an exceptional score without really knowing the material at all.

Most of the students I knew back in high school (including myself) got fantastic SAT scores. Were we very smart? Hey, I'd like to think so. Were we good test takers? Sure, that might be part of it. But the bottom line is that we figured out what the SATs are before we took them: a game, with its own rules and strategies, and clever ways to bend the former and apply the latter. Once you figure out the game for any of these standardized tests, it's not hard to score well.

#23967: MrsDoF — 05/04  at  08:41 PM
Socar, at #4, that was the first time _I_ have read the porcupine story, so you're still good.
PZ, I am glad to be reading this. As a returning to college student, I sit in class thinking that I am in way over my head, being almost twice the age of the Instructor and not knowing how to do a powerpoint presentation.
However, my high school English teacher is doing flipflops in her grave about the egregious errors in the composition papers, both with the students for writing and the Instructor with directions and grading. I cannot believe I have to sit through this to be able to put a couple letters after my name and finally get a decent job.

#23969: — 05/04  at  09:25 PM
Crap, now I can't use my GRE analytical writing score to trounce a former friend of mine who can't reason her way out of a circular argument. But I did try very hard to be factual, and I'd like to think that I write rather well for an engineer. And to think, my writing style and reasoning are nothing compared to my wordiness.

#23970: todd. — 05/04  at  09:34 PM
Not that I think highly of the SAT, but this doesn't really seem like a reason to throw the whole test out. My impression was that the essay portion was hacked on anyway. When taking the test, it's obvious that the essay section is a complete joke.

#23973: coturnix — 05/04  at  10:13 PM
Here is a blog that is entirely devoted to the SAT test, including "tricks" for playing the game:

http://triadblogs.com/rnewton

#23974: craig — 05/04  at  11:12 PM
There isn't really any such thing as an incorrect fact, is there? Alleged fact?

I am not very educated, my last real year of school having been 8th grade. I did manage to pass both ninth and tenth grades even though I never showed up and never turned in any assignments merely because I showed up for and passed the finals. (life was turbulent, school was the least of my concerns.)

I passed my 10th grade Eurpean Cultures exam by choosing the option to write an essay on a historical figure. I remembered a book I had as a little kid about the life of Bach - how his dad took him around to show off his violin playing, how he wrote his first piece at age 3, etc.

I hurried to get the hell out of there and as I was eating my lunch at a fast food restaurant it suddenly struck me - "Oh crap, that book was about Mozart!"

Didn't matter, I passed with a good grade.

#23975: craig — 05/04  at  11:14 PM
um. EurOpean.