Ai Pronunciation Assessment | AI Final Year Project Report

Complete Technical Paper Content — Ai Pronunciation Assessment

1. Executive Abstract, Problem Formulation & Background

15.ai was a free, non-commercial web application and research project that used artificial intelligence to generate text-to-speech voices of fictional characters from popular media. Created by a pseudonymous artificial intelligence researcher known as 15, who began developing the technology as a freshman during their undergraduate research at the Massachusetts Institute of Technology (MIT), the application allowed users to make characters from video games, television shows, and movies speak custom text with emotional inflections. The platform was able to generate convincing voice output using minimal training data; the name "15.ai" referenced the creator's statement that a voice could be cloned with just 15 seconds of audio. Described by AI researchers as unusually advanced for its time, with its voice synthesis outperforming contemporary models, 15.ai was one of the first mainstream applications of generative artificial intelligence during the initial stages of the AI boom. Launched in March 2020, 15.ai became an international Internet phenomenon in early 2021 when content utilizing it went viral on social media and quickly gained widespread use among Internet fandoms, such as the My Little Pony: Friendship Is Magic, Team Fortress 2, and SpongeBob SquarePants fandoms. The service featured emotional context through emojis, precise pronunciation control, and multi-speaker capabilities. Critics praised 15.ai's accessibility and emotional control but criticized its technical limitations in prosody options and non-English language support, with mixed results depending on character complexity. 15.ai is credited as the first platform to popularize AI voice cloning in memes and content creation. Voice actors and industry professionals debated 15.ai's implications, raising concerns about employment impacts, voice-related fraud, and potential misuse. In January 2022, it was discovered that a company called Voiceverse had generated voice lines using 15.ai without attribution, promoted them as the byproduct of their own technology, and sold them as non-fungible tokens (NFT) without permission. News publications universally characterized this incident as the company having stolen from 15.ai. The service went offline in September 2022 due to legal issues surrounding artificial intelligence and copyright. Its shutdown was followed by the emergence of commercial alternatives whose founders have acknowledged 15.ai's pioneering influence in the field of deep learning speech synthesis. On May 18, 2025, 15 launched 15.dev as the sequel to 15.ai; on May 28, 2026, 15 announced the indefinite closure of 15.dev to focus on founding Artist Alley, a fandom marketplace catered toward artists.

The field of speech synthesis underwent a significant transformation with the introduction of deep learning approaches. In 2016, DeepMind's publication of the WaveNet paper marked a shift toward neural network-based speech synthesis, which enabled higher audio quality via causal convolutional neural networks. Previously, concatenative synthesis—which worked by stitching together pre-recorded segments of human speech—was the predominant method for generating artificial speech, but it often produced robotic-sounding results at the boundaries of sentences. In 2018, Google AI's Tacotron 2 showed that neural networks could produce highly natural speech synthesis but required substantial training data (typically tens of hours of audio) to achieve acceptable quality. When trained on two hours of training data, the output quality degraded while still being able to maintain intelligible speech; with 24 minutes of training data, Tacotron 2 failed to produce intelligible speech. The same year saw the emergence of HiFi-GAN, a generative adversarial network (GAN)-based vocoder that improved the efficiency of waveform generation while producing high-fidelity speech, followed by Glow-TTS, which introduced a flow-based approach that allowed for both fast inference and voice style transfer capabilities. Chinese tech companies like Baidu and ByteDance also made contributions to the field by developing breakthroughs that further advanced the technology.

15.ai was conceived in 2016 as a research project in deep learning speech synthesis by a developer known as 15 (at the age of 18) during their freshman year at MIT as part of its Undergraduate Research Opportunities Program. 15 was inspired by DeepMind's WaveNet paper, with development continuing through their studies as Google AI released Tacotron 2 the following year. By 2019, they had demonstrated at MIT their ability to replicate WaveNet and Tacotron 2's results using 75% less training data than previously required. By 2020, 15 had demonstrated the ability to clone a voice using 15 seconds of audio data. The name "15.ai" is a reference to 15's statement that a voice can be perfectly cloned with as little as 15 seconds of data. 15 had originally planned to pursue a PhD based on their undergraduate research, but opted to work in the tech industry instead after their startup was accepted into the Y Combinator accelerator in 2019. After their departure in early 2020, 15 returned to their voice synthesis research and began implementing it as a web application. According to a post on X from 15, instead of using conventional voice datasets like LJSpeech that contained simple, monotone recordings, they sought out more challenging voice samples that could demonstrate the model's ability to handle complex speech patterns and emotional undertones. During this phase, 15 discovered the Pony Preservation Project, a collaborative project started by /mlp/, the My Little Pony board on 4chan. Contributors of the project had manually trimmed, denoised, transcribed, and emotion-tagged thousands of voice lines from My Little Pony: Friendship Is Magic and had compiled them into a dataset that provided ideal training material for 15.ai.

15.ai was released on March 2, 2020 as a free and non-commercial web application that did not require user registration to use, but did require the user to accept its terms of service before proceeding. At the time of its launch, the platform had a limited selection of available characters, including those from My Little Pony: Friendship Is Magic and Team Fortress 2. Users were permitted to create any content with the synthesized voices under two conditions: they had to properly credit 15.ai by including "15.ai" in any posts, videos, or projects using the generated audio; and they were prohibited from mixing 15.ai outputs with other text-to-speech outputs in the same work to prevent misrepresentation of the technology's capabilities. On March 8, 2020, Tyler McVicker of Valve News Network uploaded a YouTube video showcasing 15.ai. More voices were added to the website in the following months. In late 2020, 15 implemented a multi-speaker embedding in the deep neural network, which enabled the simultaneous training of multiple voices. Following this, the website's roster expanded from eight to over fifty characters. In addition, this implementation allowed the deep learning model to recognize common emotional patterns across different characters, even when certain emotions were missing from the characters' training data. By May 2020, the site had served over 4.2 million audio files to users. In early 2021, the application gained popularity after skits, memes, and fan content created using 15.ai went viral on Twitter, TikTok, Reddit, Twitch, Facebook, and YouTube. At its peak, the platform incurred operational costs of US$12,000 per month from AWS infrastructure needed to handle millions of daily voice generations; despite receiving offers from companies to acquire 15.ai and its underlying technology, the website remained independent and was funded out of the personal previous startup earnings of the developer.

2. Related Work, Literature Survey & Algorithmic Foundations

On January 14, 2022, 15 discovered that a blockchain-based company called Voiceverse had generated voice lines using 15.ai, falsely showcased them on Twitter as a demonstration of their own voice technology without permission or attribution, and sold them as NFTs. This came shortly after 15 had stated in December 2021 that they had no interest in incorporating NFTs into their work. A screenshot of the log files posted by 15 showed that Voiceverse had generated audio of characters from My Little Pony: Friendship Is Magic using 15.ai and pitched them up to make them sound unrecognizable, a violation of 15.ai's terms of service, which explicitly prohibited commercial use and required proper attribution. When confronted with evidence, Voiceverse stated that their marketing team had used 15.ai without proper attribution while rushing to create a demo. In response, 15 tweeted "Go fuck yourself," which went viral, amassing hundreds of thousands of retweets and likes on Twitter in support of the developer. The tweets showcasing the stolen voices were subsequently deleted.

15.ai was non-commercial, had no advertisements, generated no revenue, and operated without requiring user registration or accounts. Users were able to generate speech by inputting text and selecting a character voice, with optional parameters for emotional contextualizers and phonetic transcriptions. Each request produced three audio variations with distinct emotional deliveries. Characters available included multiple characters from Team Fortress 2 and My Little Pony: Friendship Is Magic, including the Mane Six and Derpy Hooves; GLaDOS, Wheatley, and the Sentry Turret from the Portal series; SpongeBob SquarePants; Kyu Sugardust from HuniePop, Rise Kujikawa from Persona 4; Daria Morgendorffer and Jane Lane from Daria; Carl Brutananadilewski from Aqua Teen Hunger Force; Steven Universe from Steven Universe; Sans from Undertale; Madeline and multiple characters from Celeste; the Tenth Doctor from Doctor Who; the Narrator from The Stanley Parable; and HAL 9000 from 2001: A Space Odyssey. Silent characters like Chell and Gordon Freeman were able to be selected and would emit silent audio files when any text was submitted. Characters from Undertale and Celeste did not produce spoken words but instead generated their games' distinctive beeps when text was entered.

From 2020, 15.ai had generated audio at 44.1 kHz sampling rate—higher than the 16 kHz standard used by most deep learning text-to-speech systems of that period. This higher fidelity created more detailed audio spectrograms and greater audio resolution with the tradeoff that imperfections in the synthesis were more noticeable. 15.ai processed speech using customized deep neural networks and specialized audio synthesis algorithms. While its underlying technology could produce 10 seconds of audio in less than 10 seconds of processing time (i.e. faster-than-real-time), the user experience often involved longer waits as the servers managed thousands of simultaneous requests, sometimes taking more than a minute to deliver results. Due to its nondeterministic design, 15.ai produced variations in its speech output. 15.ai introduced the concept of emotional contextualizers, which allowed users to specify the emotional tone of generated speech through guiding phrases. The emotional contextualizer functionality utilized DeepMoji to inject emotional context into synthesized speech. DeepMoji was a sentiment analysis neural network developed at the MIT Media Lab that processed emoji embeddings from 1.2 billion Twitter posts. If a prompt contained additional context (specified by a vertical bar), the additional context following the bar would be used as the emotional contextualizer. For example, if the input was Today is a great day!|I'm very sad., the selected character would speak the sentence "Today is a great day!" in the emotion one would expect from someone saying the sentence "I'm very sad." 15.ai used pronunciation data from Oxford Dictionaries API, Wiktionary, and CMU Pronouncing Dictionary, which uses ARPABET phonetic transcriptions. Users could input ARPABET transcriptions by enclosing phoneme strings in curly braces to correct mispronunciations. 15.ai's interface used color-coding to indicate pronunciation certainty and also displayed technical metrics, graphs, and comprehensive model analytics, which had included sentiment analysis and automatic improvements to the vocoder. The platform limited its prompt to 200 characters; users could combine multiple generations for longer speech sequences. Later versions of 15.ai introduced multi-speaker capabilities. Rather than training separate models for each voice, 15.ai used a unified model that learned multiple voices simultaneously through speaker embeddings: numerical representations that capture each character's unique vocal characteristics. Along with the emotional context conferred by DeepMoji, this allowed the deep learning model to learn shared patterns across different characters' emotional expressions and speaking styles, even when characters lacked examples of certain emotions in their training data.

Fan works that used 15.ai for voice acting included skits and fan animations, crossover content, recreations of viral videos, adaptations of fan fiction, music videos, and musical compositions. Team Fortress 2 content creators regularly used the platform to produce both short-form memes and complex narrative animations using Source Filmmaker.

3. Dataset Ingestion, Preprocessing & Stochastic Augmentations

PC Gamer showcased Team Fortress 2 characters redubbing a scene from Thomas the Tank Engine and arguing over a game of Among Us, while United Daily News showcased a remake of the Team Fortress 2 video "The Heavy Is Dead" and parodies of the viral "Lamar roasts Franklin" scene from Grand Theft Auto V. Kotaku showcased a version of the car dealership parody "Big Bill Hell's Cars" narrated by SpongeBob SquarePants, and Game Informer featured a Portal–Mass Effect crossover created by the author of the article herself. AUTOMATON showcased multiple Team Fortress 2 videos made using 15.ai. One used Source Filmmaker to animate a 15.ai-generated skit in which Demoman confronts Spy after discovering a drawing of his wolf fursona. Another redubbed the card reading scene from the Team Fortress short film Expiration Date, recasting it as Spy reviewing what users had created with 15.ai and growing exasperated as every example concerned a fursona, including his own. Some fan creations gained mainstream attention: a viral video that replaced Donald Trump's cameo in Home Alone 2: Lost in New York with the Heavy Weapons Guy's AI-generated voice was featured on a daytime CNN segment in January 2021.

15.ai was also regularly used within the My Little Pony fandom. In March 2020, an Equestria Daily event for generated pony voices received dozens of submissions, including episode redubs, advertisement parodies, literary recitations, song and rap covers, crossovers, and original skits. Later works featured by Equestria Daily included a full playthrough of the horror fan game Story of the Blanks using 15.ai voices, the viral Friendship is Magic–Among Us crossover animation "Among Us Struggles", and "The Tax Breaks", a fully voiced 17-minute fan-made episode.

Some users created virtual assistants using voices generated with 15.ai. In one documented project, a Portal fan used 15.ai to provide GLaDOS's voice for a customized Amazon Echo housed in a 3D printed realistic replica of the character's head, which consisted of more than 200 specially designed parts. The virtual assistant was programmed to answer selected requests with 15.ai-generated GLaDOS dialogue. Separately, Denfaminico Gamer reported a fan-created demonstration that combined 15.ai with the voice command software VoiceAttack to use GLaDOS as an assistant for controlling a computer.

AI researchers and practitioners provided more in-depth analysis of 15.ai's capabilities and limitations compared to other text-to-speech technologies of the time. Google DeepMind senior research scientist Alex Irpan wrote that when 15.ai launched in 2020, it was "arguably the highest quality voice generation model in the world" and superior to models developed by Google AI. In their explanation for why they plagiarized from 15.ai, Voiceverse wrote that "the best voice [their marketing team] could find was not [theirs] but 15.ai['s]". Rionaldi Chandraseta of Towards Data Science wrote that voice models trained on larger datasets created more convincing output with better phrasing and natural pauses, particularly for extended text. Parth Mahendra of AI Daily wrote that while the system "does a good job at accurately replicating most basic words," it struggled with more complex terms, noting that characters would "absolutely butcher the pronunciation" of certain words. Dan Ruta, the creator of a downloadable voice AI tool called xVASynth, wrote that in comparison, cloud-based tools like 15.ai had the benefit of being accessible on any device. Economist and AI commentator Tyler Cowen named 15.ai among examples of underrated talent in AI and machine learning. Chinese AI researchers also offered various technical assessments. Machine learning professor Yongqiang Li wrote that 15.ai "perfectly preserves the rhythm and characteristics of the speaker," and remarked that the application was still free despite having 5,000 people generating voices concurrently at the time of writing. Bai Feng of XinZhiYuan on QQ News highlighted the technical achievement of 15.ai's high-quality output despite using minimal training data and wrote that it was of significantly higher quality than typical deep learning text-to-speech implementations. Feng also acknowledged that while some pronunciation errors occurred due to the limited training data, it was understandable given that contemporary deep learning models typically required 40 or more hours of audio. Ji Yunyo of NetEase News called the technology behind 15.ai "remarkably efficient" but also criticized its emotional limitations, writing that the emotional expression was relatively "neutral" and that "extreme" emotions couldn't be properly synthesized, making it less suitable for not safe for work applications. Ji also wrote that while many deepfake videos required creators to extract and edit material from hours of original content for very short results, 15.ai could achieve similar or better effects with only a few dozen minutes of training data per character.

4. Neural Network Backbone, Layer Specifications & Loss Formulations

Some voice actors whose characters appeared on 15.ai have publicly shared their thoughts about the platform. In an April 2021 interview, John Patrick Lowrie—who voices the Sniper in Team Fortress 2—said that he had discovered 15.ai when a prospective intern showed him a skit she had created using AI voices of the Team Fortress 2 characters. Lowrie commented:

"The technology still has a long way to go before you really believe that these are just human beings, but I was impressed by how much [15.ai] could do. [...] You certainly don't get the delivery that you get from an actual person who's analyzed the scene, [...] but I do think that as a fan source—for people wanting to put together mods and stuff like that—that it could be fun for fans to use the voices of characters they like." He drew an analogy to synthesized music, adding:

"If you want the sound of a choir, and you want the sound of an orchestra, and you have the money, you hire a choir and an orchestra. And if you don't have the money, you have something that sounds pretty nice; but it's not the same as a choir and an orchestra." In a 2021 live broadcast on his Twitch channel, Nathan Vetterlein—the voice actor of the Scout from Team Fortress 2—listened to an AI recreation of his character's voice and commented: "It's interesting; it's all right. There's some stuff in there".

Other voice actors had mixed reactions to 15.ai's capabilities. While some industry professionals acknowledged the technical innovation, others raised concerns about the technology's implications for their profession. When voice actor Troy Baker announced his partnership with Voiceverse NFT, which had misappropriated 15.ai's technology, critics raised concerns about automated voice acting's potential reduction of employment opportunities for voice actors, risk of voice impersonation, and potential misuse in explicit content. Ruby Innes of Kotaku Australia wrote that "this practice could potentially put voice actors out of work considering you could just use their AI voice rather than getting them to voice act for a project and paying them." In her coverage of the Voiceverse controversy, Edith Wakelin-King of Checkpoint Gaming raised the concern that "this kind of technology has the potential to push voice actors out of work if it becomes easier and cheaper to use AI voices instead of working with the actor directly." While 15.ai limited its scope to fictional characters and did not reproduce voices of real people or celebrities, computer scientist Andrew Ng commented that similar technology could be used to do so, including for nefarious purposes. In his 2020 assessment of 15.ai, Ng outlined potential "enormously productive" applications of voice cloning, such as revolutionizing the use of virtual actors in Hollywood, enabling voice actors to participate in more cartoon and audiobook productions, and allowing content creators to use synthetic celebrity voices to narrate their scripts. However, he also cautioned that synthesizing a human's voice without consent raises ethical concerns and potential legal issues, and further warned that it could be maliciously exploited to impersonate private individuals.

5. Training Protocol, Convergence Behavior & Experimental Evaluation

Automatic pronunciation assessment uses computer speech recognition to determine how accurately speech has been pronounced, instead of relying on a human instructor or proctor. It is also called speech verification, pronunciation evaluation, and pronunciation scoring. This technology is used to grade speech quality, for language testing, for computer-aided pronunciation teaching (CAPT) in computer-assisted language learning (CALL), for speaking skill remediation, and for accent reduction. Pronunciation assessment is different from dictation or automatic transcription, because instead of determining unknown speech, it verifies learners' pronunciation of known word(s), often from prior transcription of the same utterance; ideally scoring the intelligibility of the learners' speech. Sometimes pronunciation assessment evaluates the prosody of the learners' speech, such as intonation, pitch, tempo, rhythm, and syllable and word stress, although those are usually not essential for being understood in most languages. Pronunciation assessment is also used in reading tutoring, for example in products from Google, Microsoft, and Amira Learning. Automatic pronunciation assessment can also be used to help diagnose and treat speech disorders such as apraxia.

Speech Pronunciation Analyzer Overview This project is designed to evaluate and improve speech pronunciation using deep learning models. It integrates multiple services for speech-to-text (ASR), phoneme alignment, and pronunciation scoring. The application provides a web interface for users to record their speech, receive feedback on their pronunciation, and track their progress. Features: Automatic Speech Recognition (ASR) using Whisper Phoneme Alignment to analyze pronunciation at the phoneme level Pronunciation Scoring for evaluating speech clarity, fluency, and accuracy Real-time Feedback for users Services The project consists of multiple services, each running on its own port. The services are: ASR Service : Converts audio input into text using Whisper. Alignment Service : Aligns the input speech to phonemes for detailed analysis. Scoring Service : Scores the pronunciation based on accuracy, fluency, and clarity. API Gateway : Handles communication between the frontend and backend services. Running the Services Prerequisites: Ensure you have Python 3.8+ and the required dependencies installed. For each service, navigate to the corresponding folder and install dependencies: cd pronunciation-evaluation/ < service > /app pip install -r requirements.txt Commands to Run Each Service: ASR Service: This service runs on port 8001. bash uvicorn app.main:app --reload --port 8001 Alignment Service: This service runs on port 8002. Note: To run the alignment service, you will need to set up a virtual environment and install specific dependencies for MFA (Montreal Forced Aligner). Steps for Alignment Service: Create a virtual environment: python -m venv venv Activate the virtual environment: On Windows: . \v env \S cripts \a ctivate On macOS/Linux: source venv/bin/activate Install the required dependencies for MFA: pip install -r requirements.txt Run the Alignment Service: uvicorn app.main:app --reload --port 8002 Scoring Service: This service runs on port 8003. uvicorn app.main:app --reload --port 8003 API Gateway: This service runs on port 8000 and serves as the main entry point for the frontend to interact with the backend services. uvicorn app.main:app --reload --port 8000 Running the Frontend: To run the frontend of the application, follow these steps: Navigate to the pronunciation-evaluation/pronunciation-frontend directory: cd pronunciation-evaluation/pronunciation-frontend Install the required dependencies: npm install Start the development server: npm run dev The frontend will be available at http://localhost:3000. API Gateway Integration: The frontend will communicate with the backend services through the API Gateway running on http://localhost:8000. Make sure all backend services are running before starting the frontend. Folder Structure pronunciation-evaluation/ ├── pronunciation-backend/ │ ├── asr-service/ │ │ └── app/ │ ├── alignment-service/ │ │ └── app/ │ ├── scoring-service/ │ │ └── app/ │ └── api-gateway/ │ └── app/ ├── pronunciation-frontend/ │ └── src/ │ └── components/ │ └── pages/ │ └── services/ └── .gitignore Conclusion This project allows users to assess and improve their pronunciation through a powerful and interactive interface. It leverages the Whisper ASR model, phoneme alignment, and deep learning-based scoring to provide real-time feedback. Ensure you have all services running and the virtual environment activated for the alignment service to properly execute the MFA alignment process. This README now includes instructions on creating and activating a virtual environment for the Alignment Service. Let me know if you ' d like any further adjustments!

This project is designed to evaluate and improve speech pronunciation using deep learning models. It integrates multiple services for speech-to-text (ASR), phoneme alignment, and pronunciation scoring. The application provides a web interface for users to record their speech, receive feedback on their pronunciation, and track their progress.

The project consists of multiple services, each running on its own port. The services are: