Robo-scoring fans like to reference a 2012 study by Mark Shermis (University of Akron) and Ben Hamner, in which computers and human scorers produced near-identical scores for a batch of essays. More recently, the former MIT professor teamed up with some students to create BABEL, a computer program that can create gibberish essays that other computer programs score as outstanding pieces of writing. He was going hand-to-hand against the 'new' SAT writing portion over a decade ago with a 5/6 essay about how Franklin Delenor Roosevelt fought the communists in 1930.

The king of robo-score debunking is Les Perelman. He pulled up a letter of recommendation he had written, replaced the student's name with words from a Criterion writing prompt, and replaced the word 'the' with 'chimpanzee.' Criterion gave the resulting essay a 6 out of 6. Back in 2007, writing instructor Andy Jones (University of California-Davis) decided to test Educational Testing Service's robo-grader Criterion. So much for open-ended questions and divergent thinking.īut the biggest problem with robo-grading continues to be the algorithm's inability to distinguish between quality and drivel.

The second is that this narrows the AI's view by saying that a good essay is one that looks a lot like these other essays. The first is that somebody has to pick the 100 exemplars, so hello again, human bias. Is punctuation not important, a little important, or super-important? You'll have to tell the program which judgment to follow, and the moment you do, you've embedded one of your personal biases into the machine.įans of robo-graders like the one in the NPR piece talk about how the AI can 'learn' what a good essay looks like by being fed a hundred or so 'good' essays. You cannot write the program without first making your own judgment about what constitutes good writing. This is particularly true for programming that scores something as subjective as writing. Only two of these are correct computer software, even Artificial Intelligence, is an encoded version of human biases. Right off the bat, Tovia Smith offers three advantages of robo-grading: cheap, quick and without human bias.

Ai essay grader