Complete Technical Paper Content — Ai Essay Scoring
1. Executive Abstract, Problem Formulation & Background
Automated Essay Scoring 2.0 with KerasNLP & LSTM Overview Welcome to our project on Automated Essay Scoring 2.0 using KerasNLP and Keras. Our goal is to develop an AI model capable of scoring student essays, improving upon existing algorithms to enhance educational outcomes. Workflow Data Preprocessing We begin by loading and preprocessing text data from CSV files containing student essays. This includes tokenization using Keras' Tokenizer and padding sequences to a maximum length for model input. Model Architecture Our model architecture utilizes a Bidirectional LSTM with attention mechanisms. This design helps capture both past and future context of words in the essays, enhancing the model's understanding of the text. GPU Setup To expedite training, we configure TensorFlow to utilize available GPUs. This setup is crucial for handling the computational demands of training large-scale natural language models efficiently. Training and Evaluation The model is trained using TensorFlow's MirroredStrategy for GPU utilization. We employ techniques such as early stopping and learning rate reduction to optimize training and prevent overfitting. Model Evaluation After training, we evaluate the model's performance using validation data. Metrics such as Mean Squared Error (MSE) are used to assess how well the model predicts essay scores. Model Architecture The above diagram illustrates the architecture of our essay scoring model. It shows the flow of data through various layers including embedding, bidirectional LSTM, attention, and dense layers for final scoring. Results and Contributions Our approach aims to significantly enhance the accuracy of automated essay scoring systems, thereby providing valuable insights into student performance. We welcome contributions and feedback to improve our model and methodology further. References Kaggle Competition
Welcome to our project on Automated Essay Scoring 2.0 using KerasNLP and Keras. Our goal is to develop an AI model capable of scoring student essays, improving upon existing algorithms to enhance educational outcomes.
We begin by loading and preprocessing text data from CSV files containing student essays. This includes tokenization using Keras' Tokenizer and padding sequences to a maximum length for model input.
Our model architecture utilizes a Bidirectional LSTM with attention mechanisms. This design helps capture both past and future context of words in the essays, enhancing the model's understanding of the text.
To expedite training, we configure TensorFlow to utilize available GPUs. This setup is crucial for handling the computational demands of training large-scale natural language models efficiently.
The model is trained using TensorFlow's MirroredStrategy for GPU utilization. We employ techniques such as early stopping and learning rate reduction to optimize training and prevent overfitting.
After training, we evaluate the model's performance using validation data. Metrics such as Mean Squared Error (MSE) are used to assess how well the model predicts essay scores.
The above diagram illustrates the architecture of our essay scoring model. It shows the flow of data through various layers including embedding, bidirectional LSTM, attention, and dense layers for final scoring.
2. Related Work, Literature Survey & Algorithmic Foundations
Our approach aims to significantly enhance the accuracy of automated essay scoring systems, thereby providing valuable insights into student performance. We welcome contributions and feedback to improve our model and methodology further.
Enhancing automated essay scoring with advanced AI techniques using Keras-NLP and LSTM, aimed at improving educational outcomes through accurate assessment of student essays.
Automated Essay Scoring System Overview The Automated Essay Scoring (AES) System is an AI-powered application designed to automatically evaluate and score essays written in educational contexts. By leveraging natural language processing (NLP) and deep learning techniques, this system mimics human judgment to provide accurate, consistent, and fair essay scoring along with constructive feedback to enhance users' writing skills. This repository contains the full project including the trained LSTM-based scoring model, data preprocessing and feature engineering workflows, and a user-friendly Flask web application interface for real-time essay evaluation. Features Automated Essay Scoring : Uses a deep learning LSTM model trained on a large essay dataset to predict scores reflecting grammar, coherence, vocabulary, and content. Grammar and Readability Analysis : Integrates grammar error detection and multiple readability metrics to provide informative feedback. User-Friendly Web Interface : Allows essay text input or DOCX file upload for scoring and displays detailed score, feedback highlights, and suggestions. Downloadable Reports : Generates color-coded feedback reports in PDF and DOCX format for offline review. Real-time Input Validation : Enforces word limits and prevents empty submissions. Advanced Linguistic Features : Incorporates tokenization, word embeddings (Word2Vec), and sequence padding for improved scoring accuracy. Multiformat Support : Handles multiple essay formats and file types. Robust Deployment : Deployed on Hugging Face Spaces using Docker container for easy access and demonstration. Project Architecture Frontend HTML/CSS/JavaScript for responsive and accessible UI Essay input area with word count and clear controls Score and feedback display with dynamic highlights Backend Flask server handling HTTP requests PyTorch for loading and running the trained LSTM model Text preprocessing (cleaning, grammar detection, readability scoring) Feedback generation via LanguageTool and custom utilities File upload and document generation (python-docx, ReportLab) Core Model LSTM neural network with 2 layers and dropout, trained with MSE loss Word2Vec embeddings as input representations Model evaluation metrics: MSE ~5.21, R² ~0.93 Technology Stack Python 3.12.1 Flask PyTorch Word2Vec (Gensim) LanguageTool (with Java OpenJDK 17) numpy, pandas python-docx, ReportLab, BeautifulSoup JavaScript / HTML / CSS Docker for containerized deployment Getting Started Prerequisites Python 3.12+ Java Runtime Environment (OpenJDK 17+) for LanguageTool Git Docker (optional, for containerized deployment) Installation Clone this repository: git clone https://github.com/yourusername/automated-essay-scoring.git cd automated-essay-scoring Create and activate a Python virtual environment: python -m venv venv source venv/bin/activate # Linux/macOS venv \S cripts \a ctivate # Windows Install required Python packages: pip install -r requirements.txt Ensure Java is installed and accessible via command line ( java -version ). Running Locally export FLASK_APP=app.py export FLASK_ENV=development flask run Open your browser at http://localhost:5000 to start using the app. Usage Submit Essay: Enter text directly or upload a DOCX file. Receive Score: The model scores the essay instantly. View Feedback: Detailed grammar and readability insights with highlighted issues. Download Report: Get your feedback in PDF or DOCX form for offline access. Limitations: Essay input is limited to 500 words for optimal performance. Project Structure / ├── app.py # Flask backend ├── model/ # Trained LSTM model and tokenizer ├── static/ # CSS, JS, images for frontend ├── templates/ # HTML templates rendered by Flask ├── utils/ # Preprocessing, feature extraction, feedback utilities ├── Dockerfile # Docker config for container deployment ├── requirements.txt # Python dependencies ├── README.md # Project documentation (this file) Testing & Validation Functional testing of essay submission, file handling, and score feedback. Model tested on diverse essays showing reliable scoring with an MSE of 5.21 and R² of 0.93. Validated downloadable report formats and UI behaviors. Edge cases such as empty input, word limit exceedance, and UI warnings are handled gracefully. Challenges Intensive model training requiring GPU resources. Integration of Java-based LanguageTool across environments. Synchronization of frontend JavaScript and backend Flask asynchronous processes. License This project is licensed under the GNU License. Acknowledgements Kaggle HP Essays Dataset used for model training. Open-source libraries: PyTorch, Flask, LanguageTool, Word2Vec. DHOLE PATIL COLLEGE OF ENGINEERING for project supervision and guidance. Thank you for exploring the Automated Essay Scoring System. Feel free to contribute.
The Automated Essay Scoring (AES) System is an AI-powered application designed to automatically evaluate and score essays written in educational contexts. By leveraging natural language processing (NLP) and deep learning techniques, this system mimics human judgment to provide accurate, consistent, and fair essay scoring along with constructive feedback to enhance users' writing skills.
This repository contains the full project including the trained LSTM-based scoring model, data preprocessing and feature engineering workflows, and a user-friendly Flask web application interface for real-time essay evaluation.
git clone https://github.com/yourusername/automated-essay-scoring.git cd automated-essay-scoring
python -m venv venv source venv/bin/activate # Linux/macOS venv \S cripts \a ctivate # Windows
/ ├── app.py # Flask backend ├── model/ # Trained LSTM model and tokenizer ├── static/ # CSS, JS, images for frontend ├── templates/ # HTML templates rendered by Flask ├── utils/ # Preprocessing, feature extraction, feedback utilities ├── Dockerfile # Docker config for container deployment ├── requirements.txt # Python dependencies ├── README.md # Project documentation (this file)
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
🚀 Instantly score essays with AI! This smart NLP-powered tool reads like a human and gives real-time feedback to level up your writing.
Automated Essay Scoring using Deep Learning and Transformer-Based Models MSc Data Science Dissertation | University of Sunderland | 2025/2026 A comprehensive evaluation of automated essay scoring systems comparing BiLSTM, BERT, and RoBERTa models with focus on performance, fairness, and explainability. 📋 Overview This repository contains the complete implementation and analysis for my MSc Data Science dissertation investigating automated essay scoring (AES) using deep learning approaches. The research compares three architectures across multiple dimensions: BiLSTM - Recurrent neural network baseline BERT - Transformer-based model RoBERTa - Optimized transformer model Key Research Questions How do transformer models (BERT/RoBERTa) compare to BiLSTM for essay scoring? Do models generalize across different datasets and populations? What fairness issues arise (prompt variation, length bias, demographic differences)? Can we explain model decisions using SHAP and attention visualization? 🎯 Key Findings Performance Results Model QWK ↑ MSE ↓ MAE ↓ Training Time BiLSTM 0.639 0.020 0.107 ~6 hours BERT 0.768 0.014 0.090 ~85 hours RoBERTa 0.761 0.013 0.088 ~85 hours 🔑 Main Finding: Transformers achieve ~20% QWK improvement over BiLSTM baseline. Fairness Analysis Model Prompt Variation ↓ Length Bias (r) ↓ BiLSTM 0.200 -0.128 BERT 0.158 -0.037 ⭐ RoBERTa 0.147 ⭐ -0.076 🔑 Main Finding: Models show complementary fairness profiles - RoBERTa achieves best prompt consistency (26.8% improvement), while BERT minimizes length bias (71% reduction). Novel Contributions Complementary Fairness Profiles : No single model dominates all fairness dimensions Performance Ceiling : Both transformers saturate near QWK 0.76-0.77 Fairness-Performance Trade-offs : Quantified across multiple equity dimensions Comprehensive Framework : Reproducible evaluation protocol for AES fairness 🚀 Quick Start Prerequisites Python 3.9 or higher 8GB RAM minimum (16GB recommended) 50GB disk space for models and datasets Installation Option 1: Automatic Setup (Recommended) Linux/Mac: chmod +x setup.sh ./setup.sh Windows: setup.bat Option 2: Manual Setup # Clone repository git clone https://github.com/YOUR_USERNAME/automated-essay-scoring-dissertation.git cd automated-essay-scoring-dissertation # Create virtual environment python -m venv venv source venv/bin/activate # Linux/Mac # OR venv \S cripts \a ctivate # Windows # Install dependencies pip install -r requirements.txt Option 3: Minimal Installation For quick testing with essential packages only: pip install -r requirements-minimal.txt Download Dataset Download ASAP dataset from Kaggle Place training_set_rel3.tsv in data/ directory 💻 Usage Training Models BiLSTM Baseline python aes_complete.py --model bilstm --epochs 50 --batch_size 32 Expected training time: ~6 hours on CPU BERT Transformer python aes_complete.py --model bert --epochs 3 --batch_size 16 Expected training time: ~85 hours on CPU, ~8 hours on GPU (V100) RoBERTa Transformer python aes_complete.py --model roberta --epochs 3 --batch_size 16 Expected training time: ~85 hours on CPU, ~8 hours on GPU (V100) 📊 Repository Structure automated-essay-scoring-dissertation/ │ ├── README.md # This file ├── requirements.txt # Full dependencies ├── requirements-minimal.txt # Essential packages only ├── requirements-dev.txt # Development tools ├── requirements-streamlit.txt # For demo app │ ├── setup.sh # Unix/Linux/Mac setup script ├── setup.bat # Windows setup script ├── .gitignore # Git ignore rules ├── LICENSE # MIT License │ ├── aes_complete.py # Main training script ├── streamlit_app.py # Interactive demo app │ ├── SETUP_GUIDE.md # Detailed installation guide ├── PACKAGES.md # Package documentation │ ├── data/ # Dataset directory │ └── README.md # Dataset instructions │ ├── results/ # Experimental results │ ├── README.md # Results summary │ ├── bilstm_cpu/ │ ├── bert_cpu/ │ └── roberta_cpu/ │ └── docs/ # Documentation ├── README.md # Docs overview └── dissertation.pdf # Full dissertation (23,709 words) 🎨 Interactive Demo Coming Soon: A Streamlit web application for interactive essay scoring! Features: Upload or paste essays for scoring Select model (BiLSTM, BERT, RoBERTa) Real-time score predictions Visualize model comparisons Explore fairness metrics Demo will be deployed to Streamlit Cloud after dissertation submission. 🎓 Academic Context Author: Theophilus Kweku Cobbah Institution: University of Sunderland Program: MSc Data Science Supervisor: Dr. Sardar Jaf Academic Year: 2025/2026 Dissertation Details Title: Automated Essay Scoring using Deep Learning and Transformer-Based Models: A Comparative Evaluation of Fairness, Explainability, and Generalisability Word Count: 23,709 Chapters: 6 (Introduction, Literature Review, Methodology, Results, Discussion, Conclusion) References: 52 academic sources 📄 Citation If you use this work, please cite: @mastersthesis { cobbah2026aes , author = { Cobbah, Theophilus Kweku } , title = { Automated Essay Scoring using Deep Learning and Transformer-Based Models: A Comparative Evaluation of Fairness, Explainability, and Generalisability } , school = { University of Sunderland } , year = { 2026 } , type = { MSc Dissertation } , url = { https://github.com/YOUR_USERNAME/automated-essay-scoring-dissertation } } 📝 License This project is licensed under the MIT License - see the LICENSE file for details. 🙏 Acknowledgments Dr. Sardar Jaf - Dissertation supervisor University of Sunderland - Research facilities and resources Hewlett Foundation - ASAP dataset creation HuggingFace - Transformers library and pre-trained models ⭐ If you find this work useful, please consider starring the repository! Last updated: December 2025
A comprehensive evaluation of automated essay scoring systems comparing BiLSTM, BERT, and RoBERTa models with focus on performance, fairness, and explainability.
This repository contains the complete implementation and analysis for my MSc Data Science dissertation investigating automated essay scoring (AES) using deep learning approaches. The research compares three architectures across multiple dimensions:
🔑 Main Finding: Models show complementary fairness profiles - RoBERTa achieves best prompt consistency (26.8% improvement), while BERT minimizes length bias (71% reduction).
# Clone repository git clone https://github.com/YOUR_USERNAME/automated-essay-scoring-dissertation.git cd automated-essay-scoring-dissertation # Create virtual environment python -m venv venv source venv/bin/activate # Linux/Mac # OR venv \S cripts \a ctivate # Windows # Install dependencies pip install -r requirements.txt
automated-essay-scoring-dissertation/ │ ├── README.md # This file ├── requirements.txt # Full dependencies ├── requirements-minimal.txt # Essential packages only ├── requirements-dev.txt # Development tools ├── requirements-streamlit.txt # For demo app │ ├── setup.sh # Unix/Linux/Mac setup script ├── setup.bat # Windows setup script ├── .gitignore # Git ignore rules ├── LICENSE # MIT License │ ├── aes_complete.py # Main training script ├── streamlit_app.py # Interactive demo app │ ├── SETUP_GUIDE.md # Detailed installation guide ├── PACKAGES.md # Package documentation │ ├── data/ # Dataset directory │ └── README.md # Dataset instructions │ ├── results/ # Experimental results │ ├── README.md # Results summary │ ├── bilstm_cpu/ │ ├── bert_cpu/ │ └── roberta_cpu/ │ └── docs/ # Documentation ├── README.md # Docs overview └── dissertation.pdf # Full dissertation (23,709 words)
Author: Theophilus Kweku Cobbah Institution: University of Sunderland Program: MSc Data Science Supervisor: Dr. Sardar Jaf Academic Year: 2025/2026
4. Neural Network Backbone, Layer Specifications & Loss Formulations
@mastersthesis { cobbah2026aes , author = { Cobbah, Theophilus Kweku } , title = { Automated Essay Scoring using Deep Learning and Transformer-Based Models: A Comparative Evaluation of Fairness, Explainability, and Generalisability } , school = { University of Sunderland } , year = { 2026 } , type = { MSc Dissertation } , url = { https://github.com/YOUR_USERNAME/automated-essay-scoring-dissertation } }
MSc Dissertation: Automated Essay Scoring using Deep Learning and Transformer-Based Models - Comparative evaluation of BiLSTM, BERT, and RoBERTa
Automatic_Essay_Scoring_AES Automatic essay scoring by extracting multiple textual features, and through just essays and target scores This project implements an Automatic Essay Scoring (AES) system using Machine Learning (ML) and Natural Language Processing (NLP) techniques. The system is designed to evaluate and score essays based on their linguistic and semantic features. The primary focus was to develop models leveraging deep learning methods such as Artificial Neural Networks (ANNs) and Long Short-Term Memory (LSTM) networks. The dataset consists of 12,977 essays from the Hewlett Foundation's Automated Essay Scoring (AES) dataset on Kaggle. Each essay includes: Text: The raw essay content. Score: The quality score assigned to the essay dataset after feature extraction uploaded to Huggingface, can be accessed from: https://huggingface.co/datasets/abdlh/Dataset_Automatic_Essay_Scoring_Essay-EssayScore_and_24_textual_features/blob/main/Updated_Processed_Data.csv Preprocessing as done in the 'new_dataset_from_old_dataset.ipynb' file Data Cleaning: Removed special characters, redundant spaces, and non-alphanumeric content. Applied case normalization (converted text to lowercase). Tokenization: Split essays into words and sentences using NLTK. Stopword Removal and Lemmatization: Removed common stopwords. Used SpaCy for lemmatization. Feature Extraction: Extracted 20+ key features such as: Linguistic Features: Word count, sentence count, average word length. Readability Scores: Flesch Reading Ease, Gunning Fog Index. Semantic Features: Sentiment analysis scores using TextBlob. POS Tagging: Distribution of parts of speech. Other Custom Features: Clause density, punctuation counts, argumentative tone, and more. Embeddings: Generated semantic embeddings using SpaCy's pre-trained language model for essay vectorization. Models and Training as in 'using_models.ipynb' and 'new_dataset_from_old_dataset.ipynb' Models: Linear Regression: Used as a baseline model with the extracted features. Artificial Neural Network (ANN): Input Layer: Matches the feature size. Hidden Layers: Two layers (64 and 32 nodes) with ReLU activation. Output Layer: Single regression neuron with linear activation. Long Short-Term Memory (LSTM): Input: Sequential embeddings for time-series analysis. Architecture: One LSTM layer (64 units) followed by dense layers. Recurrent Neural Network (RNN): Similar to the LSTM architecture but using a SimpleRNN layer. Evaluation Metrics: MSE (Mean Squared Error) MAE (Mean Absolute Error) QWK (Quadratic Weighted Kappa) Findings: Models were evaluated on an 80%-20% train-test split. A baseline LSTM (as trained in 'lstm_file.py') trained directly on essay text with fewer features outperformed others but violated the 20+ features constraint. Rest of the models trained on all 20+ features did not perform well on poor essays, though all gave great essays higher marks
Automatic essay scoring by extracting multiple textual features, and through just essays and target scores
This project implements an Automatic Essay Scoring (AES) system using Machine Learning (ML) and Natural Language Processing (NLP) techniques. The system is designed to evaluate and score essays based on their linguistic and semantic features. The primary focus was to develop models leveraging deep learning methods such as Artificial Neural Networks (ANNs) and Long Short-Term Memory (LSTM) networks.
The dataset consists of 12,977 essays from the Hewlett Foundation's Automated Essay Scoring (AES) dataset on Kaggle. Each essay includes: Text: The raw essay content. Score: The quality score assigned to the essay
Data Cleaning: Removed special characters, redundant spaces, and non-alphanumeric content. Applied case normalization (converted text to lowercase).
Tokenization: Split essays into words and sentences using NLTK. Stopword Removal and Lemmatization: Removed common stopwords. Used SpaCy for lemmatization.
5. Training Protocol, Convergence Behavior & Experimental Evaluation
Feature Extraction: Extracted 20+ key features such as: Linguistic Features: Word count, sentence count, average word length. Readability Scores: Flesch Reading Ease, Gunning Fog Index. Semantic Features: Sentiment analysis scores using TextBlob. POS Tagging: Distribution of parts of speech. Other Custom Features: Clause density, punctuation counts, argumentative tone, and more.
Embeddings: Generated semantic embeddings using SpaCy's pre-trained language model for essay vectorization.
Artificial Neural Network (ANN): Input Layer: Matches the feature size. Hidden Layers: Two layers (64 and 32 nodes) with ReLU activation. Output Layer: Single regression neuron with linear activation.
Long Short-Term Memory (LSTM): Input: Sequential embeddings for time-series analysis. Architecture: One LSTM layer (64 units) followed by dense layers.
Recurrent Neural Network (RNN): Similar to the LSTM architecture but using a SimpleRNN layer.
Metrics: MSE (Mean Squared Error) MAE (Mean Absolute Error) QWK (Quadratic Weighted Kappa)
Rest of the models trained on all 20+ features did not perform well on poor essays, though all gave great essays higher marks
Automatic essay scoring by extracting multiple textual features, and through just essays and target scores
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)