Our Research & Development
Researched by Professor Ted Briscoe’s team in Cambridge University’s Computer Laboratory, productised by iLexIR Ltd and then delivered to learners world-wide by ELiT, with input and support from leading English language organisations Cambridge Assessment English and Cambridge University Press, Write & Improve is a new kind of tool that uses natural language processing and machine learning to assess and give guidance on text it has never seen before, and to do this indistinguishably from a human examiner.
Imagine a learner was able to submit a few paragraphs of text online and, in a matter of seconds, receive an accurate grade, sentence-by-sentence feedback on its linguistic quality and useful suggestions for improvement. Imagine too that with that feedback there is motivation to try to work out how to improve the writing, rewards for progress and the opportunity to share your improvement.
This is Cambridge English Write & Improve – an online learning system, or ‘computer tutor’, to help English language learners – and it’s built on information from almost 65 million words of written English gathered over a 20-year period from essays written by real exam candidates swith 148 different mother-tongues, living in 217 different countries or territories. Each essay has been transcribed and information gathered about the learner’s age, language and grade achieved. Crucially, errors (grammar, spelling, incorrect word sequences, and so on) have been annotated so that a computer can process the natural language used by the learner. This has then been refined with nearly two million new essays which have been submitted since Write & Improve launched – and so the automated assessment engine continues to learn and improve.
“About a billion people worldwide are studying English as a further language, with a projected peak in 2050 of about two billion,” says Briscoe. “There are 300 million people actively preparing for English exams at any one time. All of them will need multiple tests during this learning process.” Language testing affects the lives of millions of people every year; a successful test result could open the door to jobs, further education and even countries.
“Imagine that a such a tool can replace one of the most routine and time-consuming tasks a teacher faces. Automating the process and still delivering individual feedback makes sense. Humans are good teachers because they show understanding of people’s problems, but machines are good at dealing with routine things and large amounts of data, seeing patterns, and giving feedback that the teacher or the learner can use. These tools can free up the teacher’s time to focus on actual teaching.”
The background to Write & Improve
Came out of Automated Language Teaching and Assessment
Partners – University of Cambridge: Cambridge Assesment English, Natural Language Processing Group, Cambridge Computer Laboratory, Applied/Corpus/Computational Linguistics, Speech Group, Engineering
Aims of the work were to research, develop and deploy next generation language learning pedagogy and technology
It is important because for English as a Further Language (EFL) there are predicted to be 2 billion EFL Learners by 2050, but not nearly enough qualified teachers (in the right places)!
EdTech is essential to meet this demand
Cambridge Learner Corpus
The building blocks for Write & improve started with robust base data and, specifically, the Cambridge Learner Corpus (CLC):
- Collaborative project between Cambridge University Press and Cambridge English Language Assessment (20+ years)
Graded texts produced by learners of English sitting Cambridge English examinations
About 40m words with 80 error types coded (60m total)
24 different exam types (including IELTS, Cambridge Preliminary (PET) / First (FCE)) – examination grades benchmarked to the CEFR
Developing grading and assessment capabilities
From this, the capability of Write & Improve was developed to deliver text-level grading and assessment, with an overall assessment of proficiency by grading the entire text
- Assess general linguistic competence
- Identify textual features which are proxies for writing competence
- Predict a grade using a weighted combination of the features
- Evaluate predicted grade
- Provide grading feedback
The other key aspect was word-level feedback methods with error detection and correction suggestions to ensure high accuracy (precision) and reasonable coverage (recall)
- Corpus-derived rules
- Error rules from the Cambridge Learner Corpus (CLC)
- Detect incorrect word sequences
- At least 90% incorrect occurrences (recurrent errors in CLC)
- Dictionary-derived rules (morphology)
- Word-level error classifier
- Precision: 90%
The system was tested in user trials:
Over 450 students (9 L1s) participated, B1(+) level
3000 submissions in total, including revisions
Over 600,000 words of text
Average response length: 200 words
Average number of revisions: 3.2
Median of number of revisions: 2
Max number of revisions: 54
Score given to the last revision is higher than that given to the initial revision in over 80% of cases
The results were:
High levels of user satisfaction
Continuous improvement for the learner
Robust correlation between human and Write & Improve grades
Conclusions of the study:
Effective feedback / grading at three different levels of granularity (text, sentence, word) and across CEFR levels / exams
Visualisation displays information in an intuitive way
Usefulness and usability of Write & Improve confirmed through user-based evaluations and uptake on web (1-6K submissions/week since March 2014)
For Write & Improve to motivate English language learners to practise and improve, it needs to be able to:
Process written scripts quickly in real time
Return level attainment (summative feedback) allowing the learner to benchmark their progress within a specified framework
Assess the relevance of the writing to the task
Deliver indirect (semi-corrective) formative feedback on specific language features in the form of error identification at sentence and word level
Offer further support to identify and understand those errors – and guidance on how they might be corrected.
All this sits within a framework which supports and motivates the learner to strive to improve their score by taking responsibility for reviewing their work.
We know from user figures that learners are able to self-motivate on Write & Improve, but self-motivation only goes so far…
We can also tell from what they write that the sort of low-stakes, no-witnesses and no-judges writing practice Write & Improve facilitates is in itself motivating. They like writing when no teacher or examiner is watching. But that also only goes so far…
And we also hold out the goal of potentially raising the learner’s CEFR level but this takes time and dedication.
We also know that learners like feedback, even if they also seem to like not having a teacher or examiner reading their work. Learners want to know how their learning is progressing.
Using the sophisticated data analytics sitting behind the tool, we can map what students do and how they respond. The tool is refined and adapted to deliver the best user experience through, for example:
Adding a wider variety of task types
Extending the scaffolding around the tool to deliver more practical hints and tips
Creating greater interaction with and support of users to celebrate their personal achievements
It is important that this constantly develops and refreshes to ensure user engagement.
R&D reference sources
The overall resource for research papers is the ALTA institute library http://www.wiki.cl.cam.ac.uk/rowiki/NaturalLanguage/ALTA
Interesting conference presentations
Cambridge English Centenary Conference, 07 Sep 2013 – Professor Ted Briscoe, Director, ALTA Institute (Automated Language Teaching and Assessment), University of Cambridge, UK – Insights from computational linguistics in teaching and assessing written English http://events.cambridgeenglish.org/cambridge-english-centenary-conference/index.html
‘Writing skills and automated feedback’ Professor Ted Briscoe’s talk at IATefl April 2017 http://iatefltalks.org/talk/writing-skills-automated-feedback
Ted Briscoe: The use of Lexical Resources in Cambridge English Write & Improve https://www.youtube.com/watch?v=AI_jhIDSMtM
Recent Papers (2016) – Most are open access — cut and paste to eg Google Scholar
C. Graham, F Nolan, A Caines, P Buttery. Toward an automated pronunciation training system for English L2 learners: theoretical and methodological issues. International Symposium on Applied Phonetics, 2016.
Andrew Caines, Christian Bentz, Calbert Graham, Tim Polzehl, Paula Buttery. Crowdsourcing a multilingual speech corpus: recording, transcription and annotation of the CrowdED Corpus. Proceedings of the Tenth International Conference on Language Resources and Evaluation, 2016.
Ronan Cummins, Meng Zhang,Ted Briscoe. Constrained Multi-Task Learning for Automated Essay Scoring. Association for Computational Linguistics, 2016.
Marek Rei, Helen Yannakoudakis. Compositional Sequence Labeling Models for Error Detection in Learner Writing. Association for Computational Linguistics, 2016.
Marek Rei, Ronan Cummins. Sentence Similarity Measures for Fine-Grained Estimation of Topical Relevance in Learner Essays. 11th Workshop on Innovative Use of NLP for Building Educational Applications, 2016.
Zheng Yuan, Ted Briscoe, Mariano Felice. Candidate re-ranking for SMT-based grammatical error correction. 11th Workshop on Innovative Use of NLP for Building Educational Applications, 2016.
Zheng Yuan, Ted Briscoe. Grammatical error correction using neural machine translation. Association for Computational Linguistics, 2016.
Ekaterina Kochmar, Ekaterina Shutova. Cross-Lingual Lexico-Semantic Transfer in Language Learning. Association for Computational Linguistics, 2016.
Dimitrios Alikaniotis, Helen Yannakoudakis, Marek Rei. Automatic Text Scoring Using Neural Networks. Association for Computational Linguistics, 2016.
Ronan Cummins, Helen Yannakoudakis, Ted Briscoe. Unsupervised Modeling of Topical Relevance in L2 Learner Text. 11th Workshop on Innovative Use of NLP for Building Educational Applications, 2016.
Text Processing Tools and Services from iLexIR Ltd
EP work by iLexIR Ltd and University of Cambridge, Computer Laboratory
Automatic grammatical analysis enabling advanced text processing in commercial applications
Interview with ALTA Institute – Write and Improve
Developing and testing a self-assessment and tutoring system
Text Readability Assessment for Second Language Learners
NAACL HLT 2016 – The Eleventh Workshop on Innovative Use of NLP for Building Educational Applications
Commentary on Write & Improve
Helpful articles setting out the premise behind Write & Improve include:
The future of English language assessment – an interview with Nick Saville of Cambridge English – Nick Saville Feb 17
Developing an ELT product based on machine learning: Write & Improve – Diane Nicholls 2017