Replaced by a robot…?

As a teacher of academic and professional writing, I often worked with students who were struggling with grammar and word-choice. Around 2014-15, students started asking whether it was worth using automatic writing evaluation (AWE) tools such as Grammarly. To test the tool, I doctored some of my own writing and ran it through a Grammarly check. I wasn’t expecting perfection, but I was shocked at how many major errors it missed and how often it flagged things that were perfectly fine, like calling out a grammatically correct bulleted list as “sentence fragments.”

Still, students kept asking whether our institution would consider paying for a Grammarly site licence. Some other universities across Canada and the U.S. had opted to do that. So in 2015, our university bought a 6-month licence that allowed a group of student peer tutors to access all of Grammarly’s features, including choices of genre and register. So they tried it, and . . .

Source: Is it time to re-think grammar-checking technology? A. Goldrick-Jones, Poster. 2019.

You can see they weren’t impressed. Grammarly not only tended to over-correct, but it couldn’t navigate the subtler rhetorical or genre-based features of documents well enough to give accurate feedback. And the tool was (still is) quite pricey. While Grammarly has a free plug-in—I use it myself, and it’s great for typos—it costs a lot per year to get the whistles and bells.

Source: Is it time to re-think grammar-checking technology? A. Goldrick-Jones, Poster. 2019.

Still, I found myself coming back to the Grammarly question. I read article after article showing how helpful AWE tools were for international students especially. Had AWE tools evolved since our study? Would we see greater accuracy, more rhetorical sophistication, and less Miss Grundyism? For a poster presentation in 2019, I compiled recent findings from other universities, teachers, and researchers about the efficacy of AWE.

There were certainly mixed reviews—in no small part because Grammarly’s educational site licences cost a hefty $1,200 per 20 users (and university budgets being as they are). But one Canadian university reported that international students appreciated the service. And while a U.S.-based writing tutor said, “I don’t think computers are even close” to processing a document’s rhetorical subtleties, he expressed high hopes for using Grammarly as a support tool. “I think the world will be all the richer in knowledge and the exchange of ideas as a result.”

Part of a 2018 Grammarly report on a doctored writing sample

In late 2018, I ran another doctored sample of my own writing through Grammarly. It correctly identified a subject-verb agreement error, a misspelling, a comma misuse, and a wrong word choice. It overlooked a missing period and apostrophe, yet called me out on correctly using commas around an appositional phrase (see picture), and it didn’t like my present-tense verbs.

Though AWE tools aren’t even close to 100% accuracy, I’m more convinced than I used to be that they have a role to play in supporting writing by giving infrequent or developing writers a leg up. This is especially noteworthy when writers—from students to professionals—are forced to work in isolation and can’t easily access human support on demand. As such tools become more reliable, they can help writers sort through basic issues before they bring in an editor or consultant.

I think editors, consultants, tutors, and teachers should be open to AWE’s possibilities. But I don’t see robots replacing us anytime soon.

_____

Have you tried AWE tools? What do you think of their reliability?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s