Open Source Llm Projects | AI Final Year Project Report

Complete Technical Paper Content — Open Source Llm Projects

1. Executive Abstract, Problem Formulation & Background

OpenArch Python implementations of modern open-source LLM architectures — written from scratch, one model at a time. This repository contains hand-written PyTorch implementations of the model architectures cataloged in Sebastian Raschka's LLM Architecture Gallery . Each model is implemented to the best of my knowledge from the original papers, technical reports, reference config.json files, and the excellent writeups by Sebastian Raschka and Machine Learning Mastery. The goal is not to compete with transformers or other production libraries. The goal is clarity and learning : a single readable file per architecture, with the structural choices (attention type, normalization, layer mix, MoE routing, positional encoding) made explicit and easy to compare side-by-side. Why this repo? Modern LLM architectures share a common skeleton but differ in dozens of small, important choices: Attention: MHA, GQA, MQA, MLA, sliding-window, linear/DeltaNet hybrids Normalization: pre-norm, post-norm, QK-Norm, sandwich norm, RMSNorm Positional encodings: RoPE, NoPE, partial RoPE, YaRN Decoder type: dense vs sparse MoE (with or without shared experts), hybrid Mamba/attention Training-time tricks: Multi-token-prediction, latent experts, gated attention Reading the official model code can be hard because production repos optimize for speed, sharding, and backward compatibility. This repo optimizes for reading . What's implemented (so far) Implementations marked ✅ are usable for forward passes; those marked 🚧 are under construction. Modality Model Status Model Size Normalization Positional Encoding Attention Mixture of Experts Text GPT-2 XL ✅ 1.5B - Absolute Multihead Attention No Llama 2 ✅ 7B RMS Norm RoPE Multihead Attention No Llama 3 ✅ 8B RMS Norm RoPE Grouped Query Attention No OLMo 2 ✅ 7B RMS Norm & QK-Norm RoPE Multihead Attention No DeepSeek R1 ✅ 671B RMS Norm & QK-Norm RoPE Multihead Latent Attention Yes Gemma 3 ✅ 27B RMS Norm & QK-Norm RoPE Grouped Query Attention with Sliding Window No Mistral 3 ✅ 24B RMS Norm RoPE Grouped Query Attention with Sliding Window No Llama 4 Maverick ✅ 400B RMS Norm RoPE Grouped Query Attention Yes Qwen 3 ✅ 4B RMS Norm & QK-Norm RoPE Grouped Query Attention No 30B-A3B RMS Norm & QK-Norm RoPE Grouped Query Attention Yes Kimi K2 ✅ 1T RMS Norm RoPE Multihead Latent Attention Yes GLM 4.5 ✅ 355B RMS Norm & QK-Norm RoPE Grouped Query Attention & Multi-Token Prediction Yes GPT-OSS ✅ 20B RMS Norm RoPE Grouped Query Attention with Sliding Window Yes Grok-2.5 🚧 270B RMS Norm RoPE Grouped Query Attention Yes Multimodal PaliGemma ✅ 3B RMS Norm RoPE Multihead Attention No Qwen3 🚧 3B RMS Norm RoPE Multihead Attention No Image Dall-e 🚧 - - - Transformer - The full target list mirrors the 72 architectures in the Architecture Gallery. Contributions toward any of them are welcome. Repository layout OpenArch/ ├── text/ │ ├── gpt2/ │ │ ├── model.py │ │ └── README.md │ ├── llama3/ │ ├── qwen3/ | ├── grok2.5/ │ └── deepseek_v3/ ├── multimodal/ │ └── pali-gemma/ │ ├── model.py │ └── README.md ├── README.md └── requirements.txt Each model lives in its own folder with respective model.py and a short README.md describing the architectural choices and references used. Contributing I am actively looking for contributors. If you enjoy reading model papers, comparing config.json files, or just want to deepen your understanding of how modern LLMs are built, this is a friendly place to start. Good first contributions: Pick an unimplemented model from the gallery and add a model.py for it Add a README.md for an existing model documenting its architectural choices Add a forward-pass test that loads the official weights and matches outputs on a few tokens Fix bugs, improve docstrings, or refactor shared components Please open an issue before starting a large piece of work so we can avoid duplicating effort. Implementations should prioritize readability over performance — this is a learning resource first. See CONTRIBUTING.md for more details. Acknowledgements This repository would not exist without the work of two outstanding educators: Sebastian Raschka — for the LLM Architecture Gallery , the Big LLM Architecture Comparison series, and the LLMs From Scratch book and codebase. The architecture diagrams, fact sheets, and side-by-side comparisons in the gallery are the primary reference behind every model in this repo. Jason Brownlee and the team at Machine Learning Mastery — for years of clear, accessible tutorials that have helped countless practitioners (myself included) build a working understanding of deep learning and transformer architectures from the ground up. Any errors in the implementations here are entirely my own. License This project is licensed under the Apache License 2.0 — see LICENSE for details. Individual model implementations follow the licenses of the original models where applicable; see each model's folder for specifics. Disclaimer These implementations are written to the best of my knowledge based on publicly available papers, technical reports, configuration files, and educational material. They are intended as a learning resource and are not affiliated with or endorsed by the original model authors. For production use, please use the official implementations or transformers .

Python implementations of modern open-source LLM architectures — written from scratch, one model at a time.

This repository contains hand-written PyTorch implementations of the model architectures cataloged in Sebastian Raschka's LLM Architecture Gallery . Each model is implemented to the best of my knowledge from the original papers, technical reports, reference config.json files, and the excellent writeups by Sebastian Raschka and Machine Learning Mastery.

The goal is not to compete with transformers or other production libraries. The goal is clarity and learning : a single readable file per architecture, with the structural choices (attention type, normalization, layer mix, MoE routing, positional encoding) made explicit and easy to compare side-by-side.

Modern LLM architectures share a common skeleton but differ in dozens of small, important choices:

Reading the official model code can be hard because production repos optimize for speed, sharding, and backward compatibility. This repo optimizes for reading .

Implementations marked ✅ are usable for forward passes; those marked 🚧 are under construction.

The full target list mirrors the 72 architectures in the Architecture Gallery. Contributions toward any of them are welcome.

OpenArch/ ├── text/ │ ├── gpt2/ │ │ ├── model.py │ │ └── README.md │ ├── llama3/ │ ├── qwen3/ | ├── grok2.5/ │ └── deepseek_v3/ ├── multimodal/ │ └── pali-gemma/ │ ├── model.py │ └── README.md ├── README.md └── requirements.txt

2. Related Work, Literature Survey & Algorithmic Foundations

Each model lives in its own folder with respective model.py and a short README.md describing the architectural choices and references used.

I am actively looking for contributors. If you enjoy reading model papers, comparing config.json files, or just want to deepen your understanding of how modern LLMs are built, this is a friendly place to start.

Please open an issue before starting a large piece of work so we can avoid duplicating effort. Implementations should prioritize readability over performance — this is a learning resource first.

This project is licensed under the Apache License 2.0 — see LICENSE for details. Individual model implementations follow the licenses of the original models where applicable; see each model's folder for specifics.

These implementations are written to the best of my knowledge based on publicly available papers, technical reports, configuration files, and educational material. They are intended as a learning resource and are not affiliated with or endorsed by the original model authors. For production use, please use the official implementations or transformers .

PyTorch implementations of modern open-source LLM architectures (Llama, Qwen, DeepSeek, Gemma, GPT-OSS, Kimi, and more) — written from scratch for readability and learning, based on Sebastian Raschka's LLM Architecture Gallery.

KalyanM45 / AI-Project-Gallery Sponsor Star 7.1k This Repository Contain All the Artificial Intelligence Projects such as Machine Learning, Deep Learning and Generative AI that I have done while understanding Advanced Techniques & Concepts. nlp-projects machine-learning-projects artificial-intelligence-projects machinelearningprojects data-science-projects machine-learning-projects-beginners deep-learning-projects computer-vision-projects ml-projects ai-projects power-bi-projects generative-ai-projects llm-project llm-projects Updated Aug 26, 2026

This Repository Contain All the Artificial Intelligence Projects such as Machine Learning, Deep Learning and Generative AI that I have done while understanding Advanced Techniques & Concepts.

InfiniteAICreations / awesome-llm-projects Star 80 😎 A list of awesome projects related to LLM awesome chatbot pr-welcome awesome-project llm llm-project Updated Feb 11, 2025

3. Dataset Ingestion, Preprocessing & Stochastic Augmentations

KalyanM45 / Multi-Agentic-Blog-Generation Sponsor Star 34 This project is a fully automated AI blogging platform built with LangGraph and Groq’s LLM. It generates and publishes technical blog posts automatically to a modern, responsive website. The system dynamically renders Markdown content without requiring complex build setups. artificial-intelligence blog-generator ai-project ai-projects blog-generation llm-project llm-prompts Updated Apr 10, 2026 Python

This project is a fully automated AI blogging platform built with LangGraph and Groq’s LLM. It generates and publishes technical blog posts automatically to a modern, responsive website. The system dynamically renders Markdown content without requiring complex build setups.

Eviltr0N / Make-AI-Clone-of-Yourself Star 30 Cloning Yourself using your whatsapp chat history and training a model on it. ai whatsapp-bot finetuning ai-project whatsapp-python whatsapp-clone llm ollama llm-project unsloth llama3 llama3-rag llama3-finetune ai-clones llama3-8b Updated Aug 14, 2024 Jupyter Notebook

hari7261 / AI-Tutor Star 22 A privacy-focused tutoring assistant that explains concepts step-by-step and generates quizzes (MCQs). Works 100% offline with Ollama (Gemma/Llama3). ai deepmind-lab deepmind ai-agent llm generative-ai ollama llm-project ai-tutor hari7261 gemma3 geema3 Updated Jul 4, 2025 Python

A privacy-focused tutoring assistant that explains concepts step-by-step and generates quizzes (MCQs). Works 100% offline with Ollama (Gemma/Llama3).

pankajtales / gen-ai-beginner-projects Star 13 Hands-on beginner-friendly Generative AI projects covering Prompt Engineering, LLM APIs, Embeddings, RAG, Vector Databases, and AI Agents. Learn Gen AI by building real-world projects. python embeddings gemini openai semantic-search ai-agents gemini-api rag embedding-vectors vector-database llm prompt-engineering generative-ai ai-projects genai langchai generative-ai-projects llm-project rag-pipeline generative-ai-agents Updated Feb 22, 2026 Python

Hands-on beginner-friendly Generative AI projects covering Prompt Engineering, LLM APIs, Embeddings, RAG, Vector Databases, and AI Agents. Learn Gen AI by building real-world projects.

sanketrs / ai-llm-project-file-structure-template Star 9 This project is a comprehensive template designed to facilitate the development and fine-tuning of Large Language Models (LLMs) ai project-template llm llm-project ai-project-resources project-file-template Updated Dec 29, 2024 Python

This project is a comprehensive template designed to facilitate the development and fine-tuning of Large Language Models (LLMs)

4. Neural Network Backbone, Layer Specifications & Loss Formulations

wahidpanda / Conversational-Chatbot-with-GROQ-with-Multi-Domain-Expertise- Star 9 Advanced AI assistant powered by Groq's lightning-fast LLMs with multi-domain knowledge and conversation analytics. python docker open-source end-to-end chatbot project ai-agents python-project ai-project huggingface streamlit ai-assistant llm langchain llm-project langchain-project Updated Apr 26, 2025 Python

Advanced AI assistant powered by Groq's lightning-fast LLMs with multi-domain knowledge and conversation analytics.

gokularaman-c / E-commerce-ai-agent Star 5 AI agent for answering SQL-based questions using LLaMA and FastAPI python nlp open-source rest-api sqlite data-analytics natural-language-to-sql fastapi ai-agent local-llm ollama llm-project llama3 ecommerce-analytics Updated Aug 7, 2025 Python

krishnaura45 / FinAI Star 5 🚀Information Retrieval & Question Answering 🔥LLMs 🌐Streamlit nlp openai llm langchain genai llm-project genai-project Updated May 7, 2026 Jupyter Notebook

mando-ai / llm_projects Star 2 LLM projects can be directly used or added to organization qna ai-project langchain-python llm-agent llm-framework llm-summarizer llm-project Updated Sep 7, 2023 Python

KalyanM45 / CareerCrawl Sponsor Star 2 This Repository contains a Python scraper that crawls company career portals, uses Large Language Models to extract insights from job descriptions, experience, work mode, and role type, and automatically filters out non-tech positions, delivering a clean, curated tech job feed. data-extraction job-board data-extractor job-portal job-search ai-agents ai-project ai-projects llm-project ai-agents-automation llm-projects Updated Jun 27, 2026 Python

This Repository contains a Python scraper that crawls company career portals, uses Large Language Models to extract insights from job descriptions, experience, work mode, and role type, and automatically filters out non-tech positions, delivering a clean, curated tech job feed.

Arjun-56 / reddit-persona-generator Star 2 A Generative AI project that builds user personas from online profile data using GPT. python user-persona generative-ai llm-project Updated Jul 15, 2025 Python

arjuntanil / NLP-CADL-Activities Star 1 CADL Activites of NLP (PMC2421A). named-entity-recognition lda text-preprocessing tf-idf-vectorization llm-project Updated Sep 30, 2025 Jupyter Notebook

5. Training Protocol, Convergence Behavior & Experimental Evaluation

chakrikaguttameedi / hiregenai-ai-hiring-intelligence Star 1 HireGenAI is a Multi-Agent AI Hiring Intelligence Platform built using LangGraph and Gemini 2.5 Flash to automate resume parsing, job matching, bias detection, and candidate ranking. It simulates a real-world intelligent recruitment workflow using structured LLM outputs and hybrid scoring logic. python nlp machine-learning artificial-intelligence gemini resume-analysis multi-agent-system ai-project streamlit llm generative-ai langgraph llm-project agentic-ai hiring-tech Updated Mar 3, 2026 Python

HireGenAI is a Multi-Agent AI Hiring Intelligence Platform built using LangGraph and Gemini 2.5 Flash to automate resume parsing, job matching, bias detection, and candidate ranking. It simulates a real-world intelligent recruitment workflow using structured LLM outputs and hybrid scoring logic.

LiteObject / llm-provider-abstraction Sponsor Star 1 A demonstration of using Adapter and Factory design patterns to create a flexible, swappable LLM provider system. This allows applications to easily switch between different LLM providers (OpenAI, Ollama, etc.) without changing business logic. python ai design-patterns factory-pattern adapter-pattern llm llm-project Updated Sep 3, 2025 Python

A demonstration of using Adapter and Factory design patterns to create a flexible, swappable LLM provider system. This allows applications to easily switch between different LLM providers (OpenAI, Ollama, etc.) without changing business logic.

lravelb / hospital-reviews Star 1 NLP project for analyzing patient satisfaction reviews using traditional machine learning and large language models (LLMs). Compares logistic regression with distilBERT on sentiment classification. nlp data-science sentiment-analysis scikit-learn healthcare transformer huggingface llm-project Updated May 26, 2025 Jupyter Notebook

NLP project for analyzing patient satisfaction reviews using traditional machine learning and large language models (LLMs). Compares logistic regression with distilBERT on sentiment classification.

rishabh1824 / AI-Powered-Resume-Analyzer Star 1 AI-powered ATS Resume Analyzer using Gemini API, Python, and semantic skill matching. flask ats python-project ai-project generative-ai resume-analyzer llm-project Updated May 5, 2026 Python

preeti28dec / LLM-Project Star 1 LLM Project llm-project Updated Mar 23, 2025 JavaScript

bytheroberto / groq-conversational-chatbot Star 0 🧠 Build an AI conversational chatbot that remembers users, manages multi-session chats, and delivers context-aware responses effortlessly. docker ai end-to-end chatbot ai-agents conversational-ai python-project ai-project huggingface streamlit session-history langchain lcel langchain-api llm-project groq-api Updated Oct 12, 2026 Jupyter Notebook