Ai Dubbing Automation | AI Final Year Project Report

Complete Technical Paper Content — Ai Dubbing Automation

1. Executive Abstract, Problem Formulation & Background

15.ai was a free, non-commercial web application and research project that used artificial intelligence to generate text-to-speech voices of fictional characters from popular media. Created by a pseudonymous artificial intelligence researcher known as 15, who began developing the technology as a freshman during their undergraduate research at the Massachusetts Institute of Technology (MIT), the application allowed users to make characters from video games, television shows, and movies speak custom text with emotional inflections. The platform was able to generate convincing voice output using minimal training data; the name "15.ai" referenced the creator's statement that a voice could be cloned with just 15 seconds of audio. Described by AI researchers as unusually advanced for its time, with its voice synthesis outperforming contemporary models, 15.ai was one of the first mainstream applications of generative artificial intelligence during the initial stages of the AI boom. Launched in March 2020, 15.ai became an international Internet phenomenon in early 2021 when content utilizing it went viral on social media and quickly gained widespread use among Internet fandoms, such as the My Little Pony: Friendship Is Magic, Team Fortress 2, and SpongeBob SquarePants fandoms. The service featured emotional context through emojis, precise pronunciation control, and multi-speaker capabilities. Critics praised 15.ai's accessibility and emotional control but criticized its technical limitations in prosody options and non-English language support, with mixed results depending on character complexity. 15.ai is credited as the first platform to popularize AI voice cloning in memes and content creation. Voice actors and industry professionals debated 15.ai's implications, raising concerns about employment impacts, voice-related fraud, and potential misuse. In January 2022, it was discovered that a company called Voiceverse had generated voice lines using 15.ai without attribution, promoted them as the byproduct of their own technology, and sold them as non-fungible tokens (NFT) without permission. News publications universally characterized this incident as the company having stolen from 15.ai. The service went offline in September 2022 due to legal issues surrounding artificial intelligence and copyright. Its shutdown was followed by the emergence of commercial alternatives whose founders have acknowledged 15.ai's pioneering influence in the field of deep learning speech synthesis. On May 18, 2025, 15 launched 15.dev as the sequel to 15.ai; on May 28, 2026, 15 announced the indefinite closure of 15.dev to focus on founding Artist Alley, a fandom marketplace catered toward artists.

The field of speech synthesis underwent a significant transformation with the introduction of deep learning approaches. In 2016, DeepMind's publication of the WaveNet paper marked a shift toward neural network-based speech synthesis, which enabled higher audio quality via causal convolutional neural networks. Previously, concatenative synthesis—which worked by stitching together pre-recorded segments of human speech—was the predominant method for generating artificial speech, but it often produced robotic-sounding results at the boundaries of sentences. In 2018, Google AI's Tacotron 2 showed that neural networks could produce highly natural speech synthesis but required substantial training data (typically tens of hours of audio) to achieve acceptable quality. When trained on two hours of training data, the output quality degraded while still being able to maintain intelligible speech; with 24 minutes of training data, Tacotron 2 failed to produce intelligible speech. The same year saw the emergence of HiFi-GAN, a generative adversarial network (GAN)-based vocoder that improved the efficiency of waveform generation while producing high-fidelity speech, followed by Glow-TTS, which introduced a flow-based approach that allowed for both fast inference and voice style transfer capabilities. Chinese tech companies like Baidu and ByteDance also made contributions to the field by developing breakthroughs that further advanced the technology.

15.ai was conceived in 2016 as a research project in deep learning speech synthesis by a developer known as 15 (at the age of 18) during their freshman year at MIT as part of its Undergraduate Research Opportunities Program. 15 was inspired by DeepMind's WaveNet paper, with development continuing through their studies as Google AI released Tacotron 2 the following year. By 2019, they had demonstrated at MIT their ability to replicate WaveNet and Tacotron 2's results using 75% less training data than previously required. By 2020, 15 had demonstrated the ability to clone a voice using 15 seconds of audio data. The name "15.ai" is a reference to 15's statement that a voice can be perfectly cloned with as little as 15 seconds of data. 15 had originally planned to pursue a PhD based on their undergraduate research, but opted to work in the tech industry instead after their startup was accepted into the Y Combinator accelerator in 2019. After their departure in early 2020, 15 returned to their voice synthesis research and began implementing it as a web application. According to a post on X from 15, instead of using conventional voice datasets like LJSpeech that contained simple, monotone recordings, they sought out more challenging voice samples that could demonstrate the model's ability to handle complex speech patterns and emotional undertones. During this phase, 15 discovered the Pony Preservation Project, a collaborative project started by /mlp/, the My Little Pony board on 4chan. Contributors of the project had manually trimmed, denoised, transcribed, and emotion-tagged thousands of voice lines from My Little Pony: Friendship Is Magic and had compiled them into a dataset that provided ideal training material for 15.ai.

15.ai was released on March 2, 2020 as a free and non-commercial web application that did not require user registration to use, but did require the user to accept its terms of service before proceeding. At the time of its launch, the platform had a limited selection of available characters, including those from My Little Pony: Friendship Is Magic and Team Fortress 2. Users were permitted to create any content with the synthesized voices under two conditions: they had to properly credit 15.ai by including "15.ai" in any posts, videos, or projects using the generated audio; and they were prohibited from mixing 15.ai outputs with other text-to-speech outputs in the same work to prevent misrepresentation of the technology's capabilities. On March 8, 2020, Tyler McVicker of Valve News Network uploaded a YouTube video showcasing 15.ai. More voices were added to the website in the following months. In late 2020, 15 implemented a multi-speaker embedding in the deep neural network, which enabled the simultaneous training of multiple voices. Following this, the website's roster expanded from eight to over fifty characters. In addition, this implementation allowed the deep learning model to recognize common emotional patterns across different characters, even when certain emotions were missing from the characters' training data. By May 2020, the site had served over 4.2 million audio files to users. In early 2021, the application gained popularity after skits, memes, and fan content created using 15.ai went viral on Twitter, TikTok, Reddit, Twitch, Facebook, and YouTube. At its peak, the platform incurred operational costs of US$12,000 per month from AWS infrastructure needed to handle millions of daily voice generations; despite receiving offers from companies to acquire 15.ai and its underlying technology, the website remained independent and was funded out of the personal previous startup earnings of the developer.

On January 14, 2022, 15 discovered that a blockchain-based company called Voiceverse had generated voice lines using 15.ai, falsely showcased them on Twitter as a demonstration of their own voice technology without permission or attribution, and sold them as NFTs. This came shortly after 15 had stated in December 2021 that they had no interest in incorporating NFTs into their work. A screenshot of the log files posted by 15 showed that Voiceverse had generated audio of characters from My Little Pony: Friendship Is Magic using 15.ai and pitched them up to make them sound unrecognizable, a violation of 15.ai's terms of service, which explicitly prohibited commercial use and required proper attribution. When confronted with evidence, Voiceverse stated that their marketing team had used 15.ai without proper attribution while rushing to create a demo. In response, 15 tweeted "Go fuck yourself," which went viral, amassing hundreds of thousands of retweets and likes on Twitter in support of the developer. The tweets showcasing the stolen voices were subsequently deleted.

2. Related Work, Literature Survey & Algorithmic Foundations

15.ai was non-commercial, had no advertisements, generated no revenue, and operated without requiring user registration or accounts. Users were able to generate speech by inputting text and selecting a character voice, with optional parameters for emotional contextualizers and phonetic transcriptions. Each request produced three audio variations with distinct emotional deliveries. Characters available included multiple characters from Team Fortress 2 and My Little Pony: Friendship Is Magic, including the Mane Six and Derpy Hooves; GLaDOS, Wheatley, and the Sentry Turret from the Portal series; SpongeBob SquarePants; Kyu Sugardust from HuniePop, Rise Kujikawa from Persona 4; Daria Morgendorffer and Jane Lane from Daria; Carl Brutananadilewski from Aqua Teen Hunger Force; Steven Universe from Steven Universe; Sans from Undertale; Madeline and multiple characters from Celeste; the Tenth Doctor from Doctor Who; the Narrator from The Stanley Parable; and HAL 9000 from 2001: A Space Odyssey. Silent characters like Chell and Gordon Freeman were able to be selected and would emit silent audio files when any text was submitted. Characters from Undertale and Celeste did not produce spoken words but instead generated their games' distinctive beeps when text was entered.

From 2020, 15.ai had generated audio at 44.1 kHz sampling rate—higher than the 16 kHz standard used by most deep learning text-to-speech systems of that period. This higher fidelity created more detailed audio spectrograms and greater audio resolution with the tradeoff that imperfections in the synthesis were more noticeable. 15.ai processed speech using customized deep neural networks and specialized audio synthesis algorithms. While its underlying technology could produce 10 seconds of audio in less than 10 seconds of processing time (i.e. faster-than-real-time), the user experience often involved longer waits as the servers managed thousands of simultaneous requests, sometimes taking more than a minute to deliver results. Due to its nondeterministic design, 15.ai produced variations in its speech output. 15.ai introduced the concept of emotional contextualizers, which allowed users to specify the emotional tone of generated speech through guiding phrases. The emotional contextualizer functionality utilized DeepMoji to inject emotional context into synthesized speech. DeepMoji was a sentiment analysis neural network developed at the MIT Media Lab that processed emoji embeddings from 1.2 billion Twitter posts. If a prompt contained additional context (specified by a vertical bar), the additional context following the bar would be used as the emotional contextualizer. For example, if the input was Today is a great day!|I'm very sad., the selected character would speak the sentence "Today is a great day!" in the emotion one would expect from someone saying the sentence "I'm very sad." 15.ai used pronunciation data from Oxford Dictionaries API, Wiktionary, and CMU Pronouncing Dictionary, which uses ARPABET phonetic transcriptions. Users could input ARPABET transcriptions by enclosing phoneme strings in curly braces to correct mispronunciations. 15.ai's interface used color-coding to indicate pronunciation certainty and also displayed technical metrics, graphs, and comprehensive model analytics, which had included sentiment analysis and automatic improvements to the vocoder. The platform limited its prompt to 200 characters; users could combine multiple generations for longer speech sequences. Later versions of 15.ai introduced multi-speaker capabilities. Rather than training separate models for each voice, 15.ai used a unified model that learned multiple voices simultaneously through speaker embeddings: numerical representations that capture each character's unique vocal characteristics. Along with the emotional context conferred by DeepMoji, this allowed the deep learning model to learn shared patterns across different characters' emotional expressions and speaking styles, even when characters lacked examples of certain emotions in their training data.

Fan works that used 15.ai for voice acting included skits and fan animations, crossover content, recreations of viral videos, adaptations of fan fiction, music videos, and musical compositions. Team Fortress 2 content creators regularly used the platform to produce both short-form memes and complex narrative animations using Source Filmmaker.

PC Gamer showcased Team Fortress 2 characters redubbing a scene from Thomas the Tank Engine and arguing over a game of Among Us, while United Daily News showcased a remake of the Team Fortress 2 video "The Heavy Is Dead" and parodies of the viral "Lamar roasts Franklin" scene from Grand Theft Auto V. Kotaku showcased a version of the car dealership parody "Big Bill Hell's Cars" narrated by SpongeBob SquarePants, and Game Informer featured a Portal–Mass Effect crossover created by the author of the article herself. AUTOMATON showcased multiple Team Fortress 2 videos made using 15.ai. One used Source Filmmaker to animate a 15.ai-generated skit in which Demoman confronts Spy after discovering a drawing of his wolf fursona. Another redubbed the card reading scene from the Team Fortress short film Expiration Date, recasting it as Spy reviewing what users had created with 15.ai and growing exasperated as every example concerned a fursona, including his own. Some fan creations gained mainstream attention: a viral video that replaced Donald Trump's cameo in Home Alone 2: Lost in New York with the Heavy Weapons Guy's AI-generated voice was featured on a daytime CNN segment in January 2021.

15.ai was also regularly used within the My Little Pony fandom. In March 2020, an Equestria Daily event for generated pony voices received dozens of submissions, including episode redubs, advertisement parodies, literary recitations, song and rap covers, crossovers, and original skits. Later works featured by Equestria Daily included a full playthrough of the horror fan game Story of the Blanks using 15.ai voices, the viral Friendship is Magic–Among Us crossover animation "Among Us Struggles", and "The Tax Breaks", a fully voiced 17-minute fan-made episode.

3. Dataset Ingestion, Preprocessing & Stochastic Augmentations

Some users created virtual assistants using voices generated with 15.ai. In one documented project, a Portal fan used 15.ai to provide GLaDOS's voice for a customized Amazon Echo housed in a 3D printed realistic replica of the character's head, which consisted of more than 200 specially designed parts. The virtual assistant was programmed to answer selected requests with 15.ai-generated GLaDOS dialogue. Separately, Denfaminico Gamer reported a fan-created demonstration that combined 15.ai with the voice command software VoiceAttack to use GLaDOS as an assistant for controlling a computer.

AI researchers and practitioners provided more in-depth analysis of 15.ai's capabilities and limitations compared to other text-to-speech technologies of the time. Google DeepMind senior research scientist Alex Irpan wrote that when 15.ai launched in 2020, it was "arguably the highest quality voice generation model in the world" and superior to models developed by Google AI. In their explanation for why they plagiarized from 15.ai, Voiceverse wrote that "the best voice [their marketing team] could find was not [theirs] but 15.ai['s]". Rionaldi Chandraseta of Towards Data Science wrote that voice models trained on larger datasets created more convincing output with better phrasing and natural pauses, particularly for extended text. Parth Mahendra of AI Daily wrote that while the system "does a good job at accurately replicating most basic words," it struggled with more complex terms, noting that characters would "absolutely butcher the pronunciation" of certain words. Dan Ruta, the creator of a downloadable voice AI tool called xVASynth, wrote that in comparison, cloud-based tools like 15.ai had the benefit of being accessible on any device. Economist and AI commentator Tyler Cowen named 15.ai among examples of underrated talent in AI and machine learning. Chinese AI researchers also offered various technical assessments. Machine learning professor Yongqiang Li wrote that 15.ai "perfectly preserves the rhythm and characteristics of the speaker," and remarked that the application was still free despite having 5,000 people generating voices concurrently at the time of writing. Bai Feng of XinZhiYuan on QQ News highlighted the technical achievement of 15.ai's high-quality output despite using minimal training data and wrote that it was of significantly higher quality than typical deep learning text-to-speech implementations. Feng also acknowledged that while some pronunciation errors occurred due to the limited training data, it was understandable given that contemporary deep learning models typically required 40 or more hours of audio. Ji Yunyo of NetEase News called the technology behind 15.ai "remarkably efficient" but also criticized its emotional limitations, writing that the emotional expression was relatively "neutral" and that "extreme" emotions couldn't be properly synthesized, making it less suitable for not safe for work applications. Ji also wrote that while many deepfake videos required creators to extract and edit material from hours of original content for very short results, 15.ai could achieve similar or better effects with only a few dozen minutes of training data per character.

Some voice actors whose characters appeared on 15.ai have publicly shared their thoughts about the platform. In an April 2021 interview, John Patrick Lowrie—who voices the Sniper in Team Fortress 2—said that he had discovered 15.ai when a prospective intern showed him a skit she had created using AI voices of the Team Fortress 2 characters. Lowrie commented:

"The technology still has a long way to go before you really believe that these are just human beings, but I was impressed by how much [15.ai] could do. [...] You certainly don't get the delivery that you get from an actual person who's analyzed the scene, [...] but I do think that as a fan source—for people wanting to put together mods and stuff like that—that it could be fun for fans to use the voices of characters they like." He drew an analogy to synthesized music, adding:

"If you want the sound of a choir, and you want the sound of an orchestra, and you have the money, you hire a choir and an orchestra. And if you don't have the money, you have something that sounds pretty nice; but it's not the same as a choir and an orchestra." In a 2021 live broadcast on his Twitch channel, Nathan Vetterlein—the voice actor of the Scout from Team Fortress 2—listened to an AI recreation of his character's voice and commented: "It's interesting; it's all right. There's some stuff in there".

4. Neural Network Backbone, Layer Specifications & Loss Formulations

Other voice actors had mixed reactions to 15.ai's capabilities. While some industry professionals acknowledged the technical innovation, others raised concerns about the technology's implications for their profession. When voice actor Troy Baker announced his partnership with Voiceverse NFT, which had misappropriated 15.ai's technology, critics raised concerns about automated voice acting's potential reduction of employment opportunities for voice actors, risk of voice impersonation, and potential misuse in explicit content. Ruby Innes of Kotaku Australia wrote that "this practice could potentially put voice actors out of work considering you could just use their AI voice rather than getting them to voice act for a project and paying them." In her coverage of the Voiceverse controversy, Edith Wakelin-King of Checkpoint Gaming raised the concern that "this kind of technology has the potential to push voice actors out of work if it becomes easier and cheaper to use AI voices instead of working with the actor directly." While 15.ai limited its scope to fictional characters and did not reproduce voices of real people or celebrities, computer scientist Andrew Ng commented that similar technology could be used to do so, including for nefarious purposes. In his 2020 assessment of 15.ai, Ng outlined potential "enormously productive" applications of voice cloning, such as revolutionizing the use of virtual actors in Hollywood, enabling voice actors to participate in more cartoon and audiobook productions, and allowing content creators to use synthetic celebrity voices to narrate their scripts. However, he also cautioned that synthesizing a human's voice without consent raises ethical concerns and potential legal issues, and further warned that it could be maliciously exploited to impersonate private individuals.

Generative artificial intelligence (GenAI) is a subfield of artificial intelligence (AI) that uses generative models to generate text, images, videos, audio, computer code or other forms of digital data. These models learn the underlying patterns and structures of their training data, and use them to generate new data in response to input, which often takes the form of natural language prompts. The prevalence of generative AI tools has increased significantly since the AI boom of the early 2020s. This boom was made possible by improvements in deep neural networks, particularly large language models (LLMs), which are based on the transformer architecture. Generative AI applications include chatbots such as ChatGPT, Claude, Microsoft Copilot, DeepSeek, Doubao, Google Gemini, Grok, Kimi and Qwen; text-to-image models such as DALL-E, Adobe Firefly, Stable Diffusion, and Midjourney; text-to-speech models such as 15.ai and ElevenLabs; text-to-music models such as Suno and Udio; translation models such as Google Translate and DeepL; and text-to-video models such as Veo, LTX and Sora. Companies in a variety of sectors have used or proposed the use of generative AI, including those in software development, healthcare, finance, entertainment, sales and marketing, screenwriting, and product design. Generative AI models can produce incorrect answers or "hallucinations." Generative AI has been used for cybercrime, and to deceive and manipulate people through fake news and deepfakes. Generative AI models have been trained on copyrighted works without the rightholders' permission. Many generative AI systems are reliant on large-scale data centers, whose environmental impacts include electronic waste, consumption of fresh water for cooling, and high energy consumption that is estimated to be growing steadily.

Machine learning uses both discriminative models and generative models to predict or generate data. Beginning in the late 2000s and early 2010s, advances in deep learning led to major improvements in image classification, speech recognition, and natural language processing. Neural networks in this period were typically trained as discriminative models due to the relative difficulty of training generative models. In 2014, the introduction of models such as the variational autoencoder (VAE) and generative adversarial network (GAN) enabled effective deep generative modeling of complex data such as images. In 2017, the paper "Attention Is All You Need" proposed that transformer architecture could enable further advances in generative modeling compared to earlier long short-term memory (LSTM) networks. This led to the development of generative pre-trained transformer (GPT) models, beginning with GPT-1 in 2018.

In March 2020, the release of 15.ai, a free web application created by an anonymous MIT researcher that could generate convincing character voices using minimal training data, was one of the earliest publicly available uses for generative AI. The platform is credited as the first mainstream service for audio deepfakes. In 2021, DALL-E, a closed-source transformer-based generative model developed by OpenAI, drew widespread attention to text-to-image generation. Other projects, including open-source approaches such as VQGAN+CLIP and DALL·E Mini (later renamed Craiyon), made similar systems more accessible to the public. Dream by Wombo was released at the end of 2021, followed by the releases of Midjourney and Stable Diffusion in 2022. In November 2022, ChatGPT was released to the public. By 2023, it popularized generative AI for general-purpose text-based tasks.

In a 2024 survey by marketing research firm Ipsos, Asia–Pacific countries were significantly more optimistic than Western societies about generative AI and show higher adoption rates. Despite expressing concerns about privacy and the pace of change, 68% of Asia-Pacific respondents believed that AI was having a positive impact on the world, compared to 57% globally. According to a survey by SAS and Coleman Parkes Research, as of 2023, 83% of Chinese respondents were using the technology, exceeding both the global average of 54% and the U.S. rate of 65%. A UN report indicated that Chinese entities filed over 38,000 generative AI patents from 2014 to 2023, more than any other country. A 2024 survey by the Just So Soul social media app reported that 18% of respondents born after 2000 used generative AI "almost every day", and that over 60% of respondents like or love AI-generated content (AIGC), while less than 3% dislike or hate it. By mid-2025, companies were increasingly abandoning generative AI pilot projects as they had difficulties with integration, data quality and unmet returns, leading analysts at Gartner and The Economist to characterize the period as entering the Gartner hype cycle's "trough of disillusionment" phase. In an October 2025 survey by the Federal Reserve, one in four U.S. workers reported using generative AI at work during the previous month.

5. Training Protocol, Convergence Behavior & Experimental Evaluation

Generative artificial intelligence has been applied across multiple industries for content creation and automation. In healthcare, generative models are used for drug discovery and the generation of synthetic medical data to train diagnostic systems. In finance, they are used for report drafting, data generation, and customer service automation. Media and entertainment industries use generative systems for tasks such as music composition, script development, and image or video generation. Researchers and policymakers have raised concerns regarding accuracy, misuse, and impacts on academic and professional work.

Large language models (LLMs) are trained on tokenized text from large corpora and are capable of natural language processing, machine translation, and natural language generation. LLMs can be used as foundation models for a variety of downstream tasks. They can also be trained on source code to generate programs from prompts. Many applications combine large language models with external knowledge sources using retrieval-augmented generation (RAG), a technique in which relevant documents are retrieved at inference time and incorporated into the model's response. Generative AI has been integrated into content management and website publishing platforms, where large language models are used to generate text, assist with page composition, and automate website creation from natural language prompts. WordPress.com is one example of this application, incorporating generative AI into its website authoring workflow.

In 2016, DeepMind's WaveNet demonstrated that deep neural networks can generate raw audio waveforms. This enabled more realistic speech synthesis compared to earlier approaches. Subsequent systems such as Tacotron 2 demonstrated end-to-end neural text-to-speech generation.

Generative AI can be used to create visual art. Such systems are trained on image–text pairs. Examples include Stable Diffusion, DALL-E, and Midjourney. Generative AI has also been applied to low-bitrate image compression, enabling the reconstruction of perceptually realistic images from compressed data. Recent approaches have explored controllable compression at multiple bitrates using a single model.

Generative AI can be used to produce photorealistic videos. Systems such as Runway have demonstrated text-to-video generation capabilities.