June 10, 2012

PRODUCTIVITY IS THE LOSS OF JOBS (via The Mother Judd):

The Algorithm Didn't Like My Essay (RANDALL STROSS, June 9, 2012, NY Times)

The standardized tests administered by the states at the end of the school year typically have an essay-writing component, requiring the hiring of humans to grade them one by one. This spring, the William and Flora Hewlett Foundation sponsored a competition to see how well algorithms submitted by professional data scientists and amateur statistics wizards could predict the scores assigned by human graders. The winners were announced last month -- and the predictive algorithms were eerily accurate. [...]

Barbara Chow, education program director at the Hewlett Foundation, says: "We had heard the claim that the machine algorithms are as good as human graders, but we wanted to create a neutral and fair platform to assess the various claims of the vendors. It turns out the claims are not hype."

If the thought of an algorithm replacing a human causes queasiness, consider this: In states' standardized tests, each essay is typically scored by two human graders; machine scoring replaces only one of the two. And humans are not necessarily ideal graders: they provide an average of only three minutes of attention per essay, Ms. Chow says.

We are talking here about providing a very rough kind of measurement, the assignment of a single summary score on, say, a seventh grader's essay, not commentary on the use of metaphor in a college senior's creative writing seminar.

Software sharply lowers the cost of scoring those essays -- a matter of great importance because states have begun to toss essay evaluation to the wayside.

"A few years back, almost all states evaluated writing at multiple grade levels, requiring students to actually write," says Mark D. Shermis, dean of the college of education at the University of Akron in Ohio. "But a few, citing cost considerations, have either switched back to multiple-choice format to evaluate or have dropped writing evaluation altogether."

As statistical models for automated essay scoring are refined, Professor Shermis says, the current $2 or $3 cost of grading each one with humans could be virtually eliminated, at least theoretically.

Posted by at June 10, 2012 9:36 AM
  

blog comments powered by Disqus
« WHICH RAISES AN OBVIOUS QUESTION...: | Main | BLUE ON BLUE: »