Member-only story
ChatGPT vs Grammarly: Who Wins the Battle of Grammatical Error Correction?
Scientists Evaluate ChatGPT Performance on Grammatical Error Correction Benchmark
Are you tired of struggling with grammatical errors in your writing? As an AI enthusiast, you might be wondering which AI tool is the best for grammatical error correction. Two popular AI writing assistants that come to mind are ChatGPT and Grammarly. But which one is better for Grammatical Error Correction (GEC)?
According to an empirical study evaluating ChatGPT1’s performance on the CoNLL2014 benchmark dataset, Grammarly outperforms ChatGPT in terms of automatic evaluation metrics. However, ChatGPT’s unique ability to go beyond one-by-one corrections by changing surface expressions and sentence structure while maintaining grammatical correctness is promising for the future of GEC.
The study found that ChatGPT tends to correct as many errors as possible, which may lead to more overcorrections. In contrast, GECToR, a state-of-the-art model, corrects only those errors it is confident about, which leaves many errors uncorrected. Grammarly, on the other hand, combines the advantages of both such that it performs more stably.
The study also showed that ChatGPT performs worse than the baseline systems in terms of the automatic evaluation metrics (e.g., F0.5 score), particularly on long sentences. However, ChatGPT goes beyond one-by-one corrections by introducing more changes to the surface expression of certain phrases or sentence structure while maintaining the grammatical correctness.
Human evaluation quantitatively demonstrated that ChatGPT produces fewer under-correction or mis-correction issues but more over-corrections. These results demonstrate the limitation of…
