Complete Technical Paper Content — Ai Genomics
1. Executive Abstract, Problem Formulation & Background
🧬 GenomeAI Next-Generation AI Framework for Genomic Disease Prediction & Variant Analysis Key Features • Clinical LIS & VCF Workspace • Architecture • AI Models • API Docs • Demo Accounts & Usage Warning RESEARCH & INVESTIGATIONAL USE ONLY — NON-DIAGNOSTIC NOTICE GenomeAI is an investigational software platform developed exclusively for computational genomics research and educational decision-support. It is not an FDA-cleared, CE-marked, or IVD-certified medical device. All patient records, medical record numbers (MRNs), and demonstration VCF panels included in this repository are entirely fictional . Computational risk predictions, automated ACMG/AMP tiers, and AI assistant insights must never be used as the sole basis for clinical diagnosis, patient management, or therapeutic intervention. Independent confirmatory testing and review by a board-certified molecular geneticist or pathologist are strictly required. 📑 Table of Contents 📌 Project Overview ✨ Key Features 🔬 Clinical LIS & VCF Variant Workspace 🏗️ Architecture 📂 Folder Structure 🔄 Workflows 🛠️ Technology Stack 🤖 AI Models 📊 Dataset & Data Pipeline 📡 API Documentation 👥 Authentication & Demo Accounts 🖼️ Screenshots ⚙️ Installation 🚀 Usage Guide & Demonstration Walkthrough 📈 Performance Benchmark 🧪 Testing & Quality Assurance ⭐ Project Highlights 🔮 Future Improvements 🧩 Challenges Faced 🎓 Learning Outcomes 🤝 Contributing 📜 License 📬 Contact & Author 🙏 Acknowledgements 📌 Project Overview The Genomic Data Challenge Modern next-generation sequencing (NGS) produces billions of nucleotide base pairs daily. Translating raw genomic data into accurate, actionable, and interpretable clinical intelligence presents two foundational bottlenecks: Biological Sequence Context : Traditional bioinformatic pipelines rely on static catalog lookups that fail to recognize complex contextual sequence motifs, non-linear regulatory patterns, and novel Single Nucleotide Variants (SNVs). Clinical Workflow Disconnect : Bioinformatic pipelines often produce isolated, massive VCF files detached from patient history, lab requisition metadata, structured prioritization rules (such as ACMG/AMP criteria), and tamper-evident audit trails. The GenomeAI Solution GenomeAI unifies cutting-edge genomic foundation models with an enterprise-grade Clinical Laboratory Information System (LIS) and VCF Variant Interpretation Workspace : Dual Analytical Engines : Genomic Deep Learning Sequence Predictor : Treats DNA sequences as a biological language, leveraging 1D Convolutional Neural Networks (1D-CNN), Bidirectional LSTMs (Bi-LSTM), and fine-tuned Genomic Transformers ( InstaDeepAI/nucleotide-transformer-v2-50m-multi-species ) with SHAP feature attribution. Clinical VCF Workspace & Prioritization Engine : Ingests standard VCF 4.2 sequence files, matches variants against curated ClinVar knowledgebases, and executes deterministic waterfall prioritization rules (R1–R10) aligned with ACMG/AMP standards. Auditable Case Management : Generates immutable case snapshots ( CASE-YYYY-NNNN ), computes SHA-256 integrity checksums for stored VCF files, records full audit history, and renders standalone, print-ready clinical reports. Contextual Genomic AI Assistant : Embeds domain-grounded conversational AI that incorporates patient clinical indications and matched variant evidence while enforcing strict non-diagnostic guardrails. Raw DNA / VCF 4.2 ──► Validation & Parsing ──► AI Prediction & ACMG Prioritization ──► Case Ledger & SHA-256 Storage ──► Standalone Clinical Report Real-World Applications Clinical Laboratory Support : Assisting molecular geneticists in triaging variants of uncertain significance (VUS) and prioritizing high-confidence pathogenic alleles. Hereditary Disease Screening : Identifying disease-associated variants across hereditary cancer (BRCA1/2), Lynch syndrome (MLH1/MSH2), familial hypercholesterolemia (LDLR), and cardiomyopathy (MYH7). Precision Medicine & Research : Accelerating biomarker discovery and cross-architecture genomic model benchmarking. Target Personas Molecular Pathologists & Geneticists : Triage VCF variant calls, review ACMG criteria, annotate findings, and generate auditable case reports. Bioinformaticians & AI Researchers : Benchmark multi-architecture deep learning models against sequence windows and evaluate SHAP feature attributions. Laboratory Directors & Quality Officers : Maintain tamper-evident case ledgers, patient directories, and verify multi-tenant isolation. ✨ Key Features Feature Category Description 🧬 VCF 4.2 File Intake & Parsing Variant Engine Strict streaming validation of standard VCF files up to 10 MiB with GRCh38/hg38 reference genome checks. 🎯 Automated ACMG/AMP Prioritization Clinical Rules Deterministic R1–R10 waterfall rule engine categorizing variants into Tier 1 (Pathogenic), Tier 2, Tier 3 (VUS), and Tier 4 (Benign). 🗂️ Clinical Patient Directory LIS Core Multi-tenant patient directory with MRN, clinical indications, demographics, and linked demonstration panels. 📑 Immutable Case Ledger LIS Core Unique sequential CASE-YYYY-NNNN identifier assignment, immutable report snapshots, and queryable history. 🔒 Tamper-Evident VCF Storage Security Ingested VCF files stored on disk with SHA-256 cryptographic hashes verified upon every download. 📄 Standalone Case Reports Reporting Clean, print-ready /report/:caseId view displaying patient demographics, variant tables, quality metrics, and disclaimers. 💬 Contextual Genomic Assistant AI Assistant Domain-grounded assistant powered by Google Gemini (with deterministic rule fallback) aware of active case context. 👥 Multi-Tenant RBAC Security Role-Based Access Control with JWT bearer tokens for Admin, Clinician, and Researcher user roles. 🧬 DNA Sequence Upload & FASTA Deep Learning Direct sequence input supporting raw strings, multiline FASTA, and case-insensitive IUPAC nucleotide handling. ✅ Sequence Validation & Tokenization Preprocessing Strict nucleotide verification [A, T, G, C, N] with position-specific error handling and integer encoding. 🔍 SNV Mutation Detection Variant Analysis SNV extraction comparing query sequences against consensus wild-types with impact scoring. 🧠 Multi-Model Inference Engine Deep Learning Switch seamlessly between 1D-CNN (< 12 ms), Bi-LSTM (~28 ms), and Nucleotide Transformer (~140 ms). 💡 Explainable AI (SHAP) XAI Batched perturbation attribution identifying top-contributing nucleotide positions across sequence windows. 📊 Dataset Analytics & Metrics Analytics Real-time GC content statistics, sequence length distributions, and multi-class probability metrics. 📄 PDF Clinical Synthesis Reporting Automated ReportLab PDF generator creating clinical summaries with probability charts and disclaimers. 🛡️ Defensive Security & Sanitization Security Enforced payload limits (10 MiB VCF, bounded sequence lengths), CORS restrictions, and SQL parameterization. 🔬 Clinical LIS & VCF Variant Workspace The Clinical LIS & VCF Variant Workspace provides an end-to-end laboratory informatics pipeline designed to simulate a real-world molecular pathology interpretation workflow: graph LR P[👤 Patient Directory] -->|Select Patient & Linked VCF| W[🔬 Variant Workspace] U[📤 Upload Custom VCF] -->|Max 10 MiB| W W -->|Syntax & Header Check| V[✅ VCF 4.2 Streaming Validator] V -->|GRCh38 Coordinates| A[🧬 ClinVar & Annotation Engine] A -->|R1-R10 Waterfall Rules| T[🎯 ACMG/AMP Tiering Engine] T -->|Display Results| R[📊 Interactive Variant Table] R -->|Consult Case Context| C[💬 Genomic AI Assistant] R -->|Save Case| L[(📑 Immutable Case Ledger)] L -->|Generate Snapshot & SHA-256| S[🗄️ Case Report Snapshot & Stored VCF] S -->|Open Direct URL| D[📄 Standalone Clinical Report] Loading 1. Patient Directory & Demo Profiles GenomeAI includes 9 pre-configured fictional clinical profiles across diverse clinical indications: Marcus Thorne ( PAT-2026-0002 ): Severe hypercholesterolemia, tendon xanthomas (Linked panel: demo_familial_hypercholesterolemia.vcf ). Eleanor Vance ( PAT-2026-0001 ): Early-onset bilateral breast cancer family history (Linked panel: demo_hereditary_cancer.vcf ). Sophia Patel ( PAT-2026-0003 ): Suspected Lynch syndrome / Colorectal screening (Linked panel: demo_lynch_syndrome.vcf ). Lucas Bennett ( PAT-2026-0004 ): Cystic fibrosis carrier screening (Linked panel: demo_cystic_fibrosis.vcf ). Aria Montgomery ( PAT-2026-0005 ): Hypertrophic cardiomyopathy screening (Linked panel: demo_cardio_myopathy.vcf ). 2. Prioritization & Waterfall Logic (R1–R10) Variants are prioritized using an automated 10-rule deterministic waterfall aligned with clinical reporting guidelines: R1 (Pathogenic & High Impact) : Known pathogenic/likely-pathogenic ClinVar annotation combined with high structural impact (frameshift, stop-gain, essential splice site) $\rightarrow$ Tier 1 (Pathogenic) . R2 (ClinVar Confirmed Pathogenic) : Consistent pathogenic submissions without conflicting benign evidence $\rightarrow$ Tier 1 (Pathogenic) . R3 (Novel High Impact in Actionable Gene) : Truncating or splice mutation in an established disease gene (e.g., BRCA1, LDLR, CFTR) $\rightarrow$ Tier 2 (Likely Pathogenic) . R4 (ClinVar Likely Pathogenic) : Multiple supportive submissions $\rightarrow$ Tier 2 (Likely Pathogenic) . R5 (Conflicting Evidence / Novel Moderate) : Inconsistent clinical significance or missense in conserved domain $\rightarrow$ Tier 3 (VUS - Variant of Uncertain Significance) . R6–R10 (Benign / Modifier / Intronic) : High population frequency (>1% gnomAD), synonymous, or deep intronic variants without splice alteration $\rightarrow$ Tier 4 (Benign / Likely Benign) . Note: Any variant with contradictory submissions or borderline quality flags automatically receives a Review High or Manual Review Required indicator. 3. Immutable Case Persistence & SHA-256 Storage When a case is saved: The system generates an official ID: CASE-YYYY-NNNN (e.g. CASE-2026-0082 ). A complete, immutable snapshot of the patient demographics, variant evaluations, prioritization tiers, and user notes is stored in the database. The associated VCF file is persisted to disk in data/vcf_storage/ and ## 🏗️ Architecture GenomeAI employs a decoupled, multi-tiered enterprise architecture with two specialized engines: Genomic Deep Learning Sequence Predictor : High-throughput inference pipeline running 1D-CNN, Bi-LSTM, and fine-tuned Transformer models with batched SHAP attributions. Clinical LIS & VCF Interpretation Engine : End-to-end variant ingestion, streaming VCF 4.2 validation, ClinVar evidence matching, deterministic ACMG/AMP prioritization, case snapshot persistence, and SHA-256 verified VCF storage. graph TD User([👤 User / Clinician / Pathologist]) -->|HTTP / SPA| Frontend[💻 React 19 Frontend - Vite / Recharts] subgraph REST API & Security Gateway Frontend -->|OAuth2 Bearer Token / REST| Backend[🚀 FastAPI Gateway - backend/main.py] Backend --> Auth[🔐 JWT Authentication & RBAC] end subgraph Engine A: Deep Learning Sequence Predictor Backend --> SeqRouter[📡 /predict & /report Routes] SeqRouter --> Val[✅ Sequence Validation & Pydantic Schema] Val --> Tok[🔢 Tokenizer - A=0, T=1, G=2, C=3, N=4] Tok --> ModelSelect{Model Selector} ModelSelect -->|1D-CNN| CNNPredictor[🧠 1D-CNN Predictor] ModelSelect -->|Bi-LSTM| LSTMPredictor[🔁 Bi-LSTM Predictor] ModelSelect -->|Transformer| TransformerPredictor[⚡ Nucleotide Transformer] CNNPredictor --> PredictEngine[🎯 Prediction Engine] LSTMPredictor --> PredictEngine TransformerPredictor --> PredictEngine PredictEngine --> SHAP[💡 Batched SHAP Perturbation Attribution] PredictEngine --> MutEngine[🔍 SNV Mutation Classifier] end subgraph Engine B: Clinical LIS & VCF Workspace Backend --> VCFRouter[📡 /variants, /patients, /cases Routes] VCFRouter --> VCFVal[✅ VCF 4.2 Streaming Header & Syntax Validator] VCFVal --> Annotator[🧬 ClinVar & ACMG Prioritization Engine] Annotator --> Waterfall[🎯 R1-R10 Waterfall Classification] Annotator --> StorageService[🔒 VCF Storage Service - SHA-256 Hashing] Backend --> ChatRouter[📡 /chat Route] ChatRouter --> ChatService[💬 Contextual Assistant - Gemini / Grounded Fallback] end subgraph Data & Storage Tier PredictEngine --> SQLiteHist[(🗄️ Prediction History DB)] PredictEngine --> PDFGen[📄 ReportLab PDF Generator] Waterfall --> LISDB[(🗄️ PostgreSQL / Supabase / SQLite LIS DB)] StorageService --> DiskStorage[💾 Persistent VCF Storage - data/vcf_storage/] end PDFGen -->|Streaming PDF Bytes| Frontend LISDB -->|Patient, Case & Ledger Data| Frontend DiskStorage -->|SHA-256 Verified VCF| Frontend Loading 📂 Folder Structure GenomeAI/ ├── backend/ │ ├── main.py # FastAPI server entry point, CORS, routers & startup seed │ ├── api/ │ │ ├── auth_routes.py # OAuth2 JWT login & user authentication │ │ ├── case_routes.py # Case persistence, ledger history, report snapshots, VCF download │ │ ├── chat_routes.py # Domain-grounded contextual AI chatbot API │ │ ├── patient_routes.py # Patient directory, clinical indications, demo panel links │ │ ├── routes.py # Deep learning inference routes (/predict, /report, /benchmark) │ │ └── variant_routes.py # VCF streaming validation, annotation, ACMG tiering, demo fetch │ ├── ai/ # Machine learning model training scripts │ │ ├── cnn_model.py # 1D-CNN architecture & Keras training loop │ │ ├── lstm_model.py # Bi-LSTM architecture & Keras training loop │ │ ├── transformer_train.py # PyTorch fine-tuning loop for Nucleotide Transformer │ │ ├── transformer_dataset.py # PyTorch Dataset class │ │ ├── transformer_utils.py # Epoch training & validation utilities │ │ ├── data_loader.py # Training dataset splitter & class weighting │ │ ├── class_weights.py # Balanced class weight compute │ │ ├── evaluate.py # Classification metrics & confusion matrix │ │ └── metrics.py # Training loss/accuracy curve graph generator │ ├── database/ # LIS persistence layer │ │ ├── database.py # SQLAlchemy session factory (PostgreSQL / SQLite fallback) │ │ ├── models.py # Patient, Case, User, Prediction ORM entities │ │ ├── crud.py # Thread-safe database transactions & query filters │ │ └── migrations/ # Alembic database migration scripts (versions 001, 002) │ ├── demo_data/ # Curated demonstration VCF panels (Fictional GRCh38) │ │ ├── demo_cardio_myopathy.vcf │ │ ├── demo_cystic_fibrosis.vcf │ │ ├── demo_familial_hypercholesterolemia.vcf │ │ ├── demo_hereditary_cancer.vcf │ │ └── demo_lynch_syndrome.vcf │ ├── predictor/ # Single-sample inference predictors │ │ ├── predictor.py # Master predictor facade │ │ ├── cnn_predictor.py # Optimized 1D-CNN tensor caller │ │ ├── lstm_predictor.py # Optimized Bi-LSTM tensor caller │ │ └── transformer_predictor.py# PyTorch Transformer inference loader │ ├── preprocessing/ # Genome index & sequence extraction │ │ ├── dataset_builder.py # Master dataset preparation script │ │ ├── dna_tokenizer.py # Base tokenization rules │ │ ├── genome_loader.py # FASTA genome reader │ │ └── sequence_extractor.py # Flanking sequence extractor around variant IDs │ ├── services/ # Business logic & domain services │ │ ├── analytics_service.py # GC content & dataset analytics calculator │ │ ├── benchmark_service.py # Multi-model benchmarking suite │ │ ├── chat_service.py # Grounded AI assistant with ClinVar context & fallback │ │ ├── explainability_service.py # Batched SHAP feature attribution │ │ ├── model_loader.py # Lazy model loader with error handling │ │ ├── mutation_analysis.py # Point mutation detector & classifier │ │ ├── prediction_history.py # SQLite persistence & stats aggregator │ │ ├── report_generator.py # ReportLab PDF report generator │ │ ├── variant_annotation_service.py # VCF 4.2 parser, ClinVar matcher & R1-R10 tiering │ │ └── vcf_storage_service.py # Persistent disk storage with SHA-256 hashing │ └── utils/ # Helper utilities │ ├── disease_mapper.py # Integer label to disease name lookup │ └── tokenizer.py # Shared token mapping & string sanitization ├── data/ # Runtime persistence directory │ └── vcf_storage/ # Managed VCF file store (SHA-256 keyed) ├── datasets/ # Genomic data storage │ ├── raw/ # ClinVar & NCBI raw annotations │ └── processed/ # Curated & tokenized CSV datasets ├── frontend/ # React 19 Frontend Web Application │ ├── src/ │ │ ├── api/ # Axios API client (/api/client.js) │ │ ├── components/ # Reusable UI cards, navbars, & skeletons │ │ ├── pages/ # Application views │ │ │ ├── AdminDashboard.jsx # System overview, LIS statistics, and service health │ │ │ ├── CaseHistoryPage.jsx # Auditable case ledger, filter, search, VCF download │ │ │ ├── ClinicalPredictionPage.jsx # Clinician DNA sequence inference view │ │ │ ├── PatientDatabasePage.jsx # Multi-tenant patient directory & linked VCF launch │ │ │ ├── ResearchBenchmarkPage.jsx # Multi-model comparative benchmarking │ │ │ ├── VariantWorkspacePage.jsx # Interactive VCF intake, tiering, & AI assistant │ │ │ └── VcfCaseReportPage.jsx # Standalone clinical diagnostic report view (/report/:caseId) │ │ ├── styles/ # Modular CSS stylesheets │ │ ├── App.jsx # Client router, navigation, and protected routes │ │ └── main.jsx # React DOM entry point │ ├── package.json # Frontend dependencies │ └── vite.config.js # Vite dev server configuration ├── reports/ # Training loss/accuracy plot exports ├── scripts/ # Dataset indexing helper scripts ├── tests/ # Automated Pytest suite (158 passing tests) │ ├── test_chat_service.py │ ├── test_patient_case_persistence.py │ ├── test_phase3_e2e_reliability.py │ ├── test_variant_annotator.py │ ├── test_variant_api.py │ └── test_vcf_parser.py ├── trained_models/ # Model checkpoints (.keras / .pth / .db) ├── .gitignore # Git exclusion rules ├── alembic.ini # Database migration configuration ├── LICENSE # MIT License ├── README.md # Comprehensive project documentation ├── requirements.txt # Python backend dependencies └── start.py # Single-command parallel application runner 🔄 Workflows Workflow 1: DNA Sequence Deep Learning Inference & Explainability sequenceDiagram autonumber actor User as Client / Researcher participant App as React Frontend participant API as FastAPI Backend participant Engine as Model Engine participant SHAP as SHAP Attributor participant DB as SQLite Storage participant PDF as ReportLab PDF Generator User->>App: Enter DNA Sequence (201 bp) & Select Model (e.g., CNN) App->>API: POST /predict/extended { sequence, model } API->>API: Validate input length & sanitize sequence API->>Engine: Run single-sample inference Engine-->>API: Return disease probabilities & confidence API->>SHAP: Compute batched perturbation feature importance SHAP-->>API: Return top influential sequence regions API->>DB: Persist prediction record in SQLite API-->>App: Return complete JSON response App->>User: Display Disease Risk, Confidence, & SHAP Charts opt Export Report User->>App: Click "Download PDF Report" App->>API: POST /predict/report { sequence, model, patient_name } API->>PDF: Generate Clinical Summary PDF PDF-->>API: Stream PDF Binary Stream API-->>App: Download PDF file (genomeai_prediction_report.pdf) end Loading Workflow 2: Clinical VCF Variant Ingestion, Prioritization & Case Ledger sequenceDiagram autonumber actor Clinician as Pathologist / Clinician participant App as React Frontend participant API as FastAPI Backend participant Annotator as Annotation Service participant Storage as VCF Storage Service participant DB as LIS Database participant Chat as Genomic AI Assistant Clinician->>App: Open Patient Directory & Select Patient (e.g. PAT-2026-0002) Clinician->>App: Click "Analyze Variant Profile" (launches Variant Workspace) App->>API: GET /api/variants/demo?panel=demo_familial_hypercholesterolemia.vcf API-->>App: Pre-load demo VCF text Clinician->>App: Click "Run Variant Pipeline" App->>API: POST /api/variants/annotate (Multipart VCF + Patient ID) API->>Annotator: Parse VCF 4.2 header, CHROM, POS, REF, ALT API->>Annotator: Match against ClinVar & apply R1-R10 Waterfall Rules Annotator-->>API: Annotated variants, ACMG Tiers, Quality Metrics API-->>App: Return structured JSON results App->>Clinician: Render interactive variant table, tiers & review flags opt Consult Contextual AI Clinician->>App: Ask "Why is LDLR c.662A>G classified as Tier 1?" App->>API: POST /api/chat/message (Prompt + active case context) API->>Chat: Ground query with ClinVar evidence & patient indications Chat-->>API: Response with citations & mandatory non-diagnostic disclaimers API-->>App: Display assistant insights end Clinician->>App: Click "Save Case to History" App->>API: POST /api/cases (Patient ID, Variants, Notes, VCF Payload) API->>Storage: Persist VCF file to disk & calculate SHA-256 hash API->>DB: Store immutable case snapshot & assign CASE-YYYY-NNNN DB-->>API: Case confirmed (e.g. CASE-2026-0082) API-->>App: Return Case ID & Snapshot Clinician->>App: Click "View Report" -> Redirect to /report/:caseId App->>API: GET /api/cases/:caseId (Fetch snapshot) API-->>App: Return immutable case record App->>Clinician: Display clean, print-ready standalone diagnostic report Loading 🛠️ Technology Stack Layer Technologies Used Purpose Frontend UI React 19, Vite, Lucide React Modern SPA frontend framework & UI iconography. Data Visualization Recharts, Framer Motion Dynamic disease probability charts & micro-animations. Backend REST API FastAPI, Uvicorn, Pydantic v2 High-performance asynchronous REST API & schema validation. Deep Learning PyTorch 2.2+, TensorFlow 2.16+ Foundation Transformer models & 1D-CNN / Bi-LSTM networks. NLP & Transformers HuggingFace transformers Pretrained InstaDeepAI/nucleotide-transformer-v2-50m-multi-species . Bioinformatics PyFAIDX, Pandas, NumPy, Scikit-Learn FASTA genome indexing, sequence slicing, & metrics evaluation. Database Supabase PostgreSQL / SQLAlchemy / Alembic Serverless cloud PostgreSQL database (with SQLite local fallback). Document Generation ReportLab 4.1+ Dynamic vector PDF medical report generation. Environment Python 3.10+, PowerShell, Git Development runtime & version control. 🤖 AI Models 🧠 1D Convolutional Neural Network (1D-CNN) [Click to expand] Why it is used : 1D Convolutions are exceptionally effective at detecting short, localized sequence motifs (such as transcription factor binding sites or splice junctions) regardless of their exact position in the sequence window. Architecture : Embedding Layer : Input dimension 5 (A, T, G, C, N) -> Dense Vector 8. Conv1D (64 filters, kernel=5) + Batch Normalization + MaxPool1D(2) + Dropout(0.20). Conv1D (128 filters, kernel=3) + Batch Normalization + MaxPool1D(2) + Dropout(0.30). Conv1D (256 filters, kernel=3) + GlobalMaxPooling1D() + Dropout(0.40). Dense (256 units, L2 regularized) + Softmax Output (8 classes). Advantages : Blazing fast inference (< 10 ms), low memory footprint, highly effective for localized variant motif detection. Limitations : Restricted receptive field; cannot easily capture distant interactions between sequence ends. 🔁 Bidirectional LSTM (Bi-LSTM) [Click to expand] Why it is used : Recurrent architectures maintain hidden memory states across positions, processing nucleotides in both forward and reverse directions to model long-range sequential context. Architecture : Embedding Layer : Input dimension 5 -> Dense Vector 16. Stacked LSTM Layer 1 : 64 units (return_sequences=True). Stacked LSTM Layer 2 : 32 units. Dense Layer : 64 ReLU units -> Softmax Output (8 classes). Advantages : Captures sequential dependencies and directionality across the entire nucleotide window. Limitations : Sequential processing cannot be easily parallelized on GPU; slightly slower inference times. ⚡ Fine-Tuned Nucleotide Transformer (PyTorch) [Click to expand] Why it is used : Leverages transfer learning from InstaDeep's 50M parameter foundation model pre-trained on multi-species genomic assemblies to learn complex contextual representations. Architecture : Base Model : InstaDeepAI/nucleotide-transformer-v2-50m-multi-species encoder. Classification Head : 8-class linear sequence classification head fine-tuned with AdamW ( lr=2e-4 ) and linear warmup schedule. Advantages : Superior accuracy on complex variants; understands deep evolutionary sequence patterns. Limitations : Higher computational requirement (~50M parameters); requires GPU acceleration for optimal performance. 📊 Dataset & Data Pipeline GenomeAI utilizes a curated genomic dataset built from ClinVar pathogenic variant records mapped against the NCBI GRCh38.p14 human reference assembly. NCBI GRCh38.p14 Genome + ClinVar Variants ──► Sequence Extractor (201-bp window) ──► Mutated Variant Generation ──► Integer Tokenization Dataset Statistics Metric Value Total Curated Samples 68,527 variants Target Disease Classes 8 major conditions Window Length Exactly 201 nucleotides (variant centered at position 101) Mean GC Content 42.5% Nucleotide Alphabet A , T , G , C , N (Unknown/Padded) Class Label Distribution Label Target Condition Sample Count Percentage 0 Breast Cancer 12,450 18.16% 1 Lung Cancer 10,820 15.79% 2 Alzheimer's Disease 9,140 13.34% 3 Parkinson's Disease 8,650 12.62% 4 Leukemia 7,980 11.64% 5 Type 2 Diabetes 7,210 10.52% 6 Ovarian Cancer 6,430 9.38% 7 Colorectal Cancer 5,847 8.53% 📡 API Documentation Interactive OpenAPI Swagger docs are available at http://127.0.0.1:8000/docs and ReDoc at http://127.0.0.1:8000/redoc . 1. Clinical LIS & VCF Workspace Endpoints Endpoint Method Role Description /api/auth/token POST Public Authenticates credentials (OAuth2 form-data) and issues JWT access token. /api/auth/me GET Authenticated Retrieves current authenticated user identity, role, and tenant scopes. /api/patients GET Clinician / Admin Lists registered patient directory with pagination, search, and clinical indication filter. /api/patients/{id} GET Clinician / Admin Retrieves specific patient dossier, metadata, and linked demonstration VCF panel. /api/patients POST Clinician / Admin Registers a new clinical patient record with MRN, DOB, gender, and clinical history. /api/variants/validate POST Clinician / Admin Performs fast streaming header syntax, reference genome (GRCh38), and format checks. /api/variants/annotate POST Clinician / Admin Ingests VCF (multipart upload up to 10 MiB), matches ClinVar database, and applies R1–R10 ACMG tiering. /api/variants/demo GET Clinician / Admin Retrieves pre-seeded demonstration panel text by panel filename. /api/cases POST Clinician / Admin Persists case ledger entry ( CASE-YYYY-NNNN ), records immutable snapshot, and hashes VCF with SHA-256. /api/cases GET Clinician / Admin Queries paginated, auditable case history ledger with status and search filters. /api/cases/{case_id} GET Clinician / Admin Retrieves complete immutable case report snapshot for /report/:caseId . /api/cases/{case_id}/download-vcf GET Clinician / Admin Verifies SHA-256 checksum and streams the original uploaded VCF file. /api/chat/message POST Authenticated Domain-grounded conversational genomic assistant evaluating active case variants and indications. 2. Genomic Deep Learning Sequence Predictor Endpoints Endpoint Method Role Description /health GET Public Returns API health status, database connection state, and loaded neural model weights. /predict POST Authenticated Runs single-sample inference across selected model (CNN, LSTM, Transformer). /predict/extended POST Authenticated Full diagnostic suite: prediction + mutation classification + batched SHAP explainability. /predict/report POST Authenticated Synthesizes and streams dynamic ReportLab vector PDF diagnostic report. /benchmark GET Authenticated Retrieves comparative latency, accuracy, and F1 benchmarks across all architectures. /analytics GET Authenticated Returns dataset GC distribution, sequence length statistics, and class proportions. /history GET Authenticated Returns paginated deep learning sequence inference audit records from SQLite. 📄 Example API Request & Response (POST /api/variants/annotate ) [Click to expand] POST Request : http://localhost:8000/api/variants/annotate Multipart Form Data: file=@demo_familial_hypercholesterolemia.vcf , patient_id=PAT-2026-0002 JSON Response (200 OK) : { "total_variants" : 1 , "annotated_variants" : [ { "chromosome" : " chr19 " , "position" : 11200000 , "reference" : " A " , "alternate" : " G " , "gene" : " LDLR " , "consequence" : " missense_variant " , "clinvar_significance" : " Pathogenic " , "acmg_tier" : " TIER_1_PATHOGENIC " , "priority_score" : 95.0 , "review_flag" : " REVIEW_HIGH " , "rule_applied" : " R1_PATHOGENIC_HIGH_IMPACT " , "filter" : " PASS " } ], "quality_summary" : { "pass_count" : 1 , "fail_count" : 0 , "mean_depth" : 48.5 , "ts_tv_ratio" : 2.1 }, "tier_summary" : { "TIER_1_PATHOGENIC" : 1 , "TIER_2_LIKELY_PATHOGENIC" : 0 , "TIER_3_VUS" : 0 , "TIER_4_BENIGN" : 0 } } 👥 Authentication & Demo Accounts GenomeAI provides built-in multi-tenant Role-Based Access Control (RBAC). Upon initial backend startup, the database is automatically seeded with default development accounts: Role Email Password Permissions & Intended Workflow 🛡️ Admin admin@genomeai.lab admin123 System oversight, user administration, service health monitors, full access across all LIS and AI modules. 🩺 Clinician doctor@genomeai.lab doctor123 Patient Directory, VCF Variant Workspace, Prioritization Engine, Case Persistence, and Standalone Reports. 🔬 Researcher researcher@genomeai.lab research123 Deep Learning Sequence Predictor, SHAP Explainability Heatmaps, Benchmarking Suite, and Dataset Analytics. Note All accounts are initialized with password hashes generated using bcrypt and authenticated via standard OAuth2 JWT bearer tokens ( Authorization: Bearer <token> ). In local development mode, test sessions remain cached in the browser's localStorage . 🖼️ Screenshots Page / Dashboard Visual Placeholder Home Page (Placeholder: [GenomeAI Modern Hero Section & Particle Visualizer]) DNA Prediction Page (Placeholder: [Sequence Input Form, Model Switcher, & Confidence Gauge]) Mutation Analysis Page (Placeholder: [Variant Alignment View & Impact Classification Cards]) SHAP Explainability View (Placeholder: [Position-Wise Feature Importance Heatmap]) Dataset Analytics Dashboard (Placeholder: [Recharts GC Content & Disease Distribution Graphs]) Clinical PDF Report (Placeholder: [ReportLab Formatted Downloadable Medical Report PDF]) 🚀 Quick Start & Local Development Workflow Running the complete enterprise application—including React frontend, FastAPI APIs, PDF report generation, and database audit logs—is straightforward: 🚀 1. Launching GenomeAI (One-Command Startup) Run either of the following single commands from the root project directory: # Option A: NPM npm start # Option B: Python python start.py # or python run.py This single command automatically launches both services in parallel: FastAPI Backend : http://127.0.0.1:8000 (API Docs at http://127.0.0.1:8000/docs ) React Frontend : http://localhost:5173 (Vite dev server with /api proxying) 🛑 2. Stopping GenomeAI Simply press Ctrl+C in your terminal window. All backend and frontend processes will stop automatically. 🛠️ 3. Independent Component Execution Developers can run components independently: FastAPI Backend : .\venv\Scripts\Activate.ps1 python - m uvicorn backend.main:app -- host 127.0 . 0.1 -- port 8000 -- reload React Frontend : cd frontend npm run dev 🐳 Docker Deployment GenomeAI includes production-ready Docker support. You can spin up the entire application (React Frontend + FastAPI Backend) using a single command: docker compose up -- build ⚙️ Installation Prerequisites Python : 3.10 or higher (Python 3.11 recommended) Node.js : v18.0 or higher (for React 19 Frontend) Git Step 1: Clone Repository git clone https: // github.com / SuhaniKeni / GenomeAI.git cd GenomeAI Step 2: Set Up Virtual Environment # Windows PowerShell python - m venv venv .\venv\Scripts\Activate.ps1 Step 3: Install Backend Dependencies pip install - r requirements.txt Step 4: Configure Environment Variables Copy the example environment configuration template: cp .env.example .env Edit .env as needed: DATABASE_URL : Connection string for PostgreSQL / Supabase. Leave blank or commented out to use the automatic thread-safe SQLite fallback ( trained_models/genomeai_lis.db ). JWT_SECRET_KEY : Random 256-bit secret string for signing OAuth2 session tokens. GEMINI_API_KEY : (Optional) Google Gemini API key for live LLM responses in the Genomic Assistant. If omitted, the system seamlessly operates in deterministic rule-based fallback mode. Step 5: Run Database Migrations Apply database schema migrations to initialize tables, indexes, and constraints: alembic upgrade head Step 6: Install Frontend Dependencies cd frontend npm install cd .. 🚀 Usage Guide & Demonstration Walkthrough Option A: Launch Everything with One Command (Recommended) Run either of the following from the repository root: python start.py # or npm start Both the FastAPI backend ( http://127.0.0.1:8000 ) and React frontend ( http://localhost:5173 ) will launch concurrently. Option B: Launch Services Independently FastAPI REST Backend : .\venv\Scripts\Activate.ps1 python - m uvicorn backend.main:app -- host 127.0 . 0.1 -- port 8000 -- reload React Frontend : cd frontend npm run dev 🧪 Step-by-Step Clinical Demonstration Walkthrough Follow this 9-step script to evaluate the full end-to-end clinical workflow using pre-configured demonstration fixtures: Log In to the Platform : Navigate to http://localhost:5173/admin (or click Sign In in the navigation bar). Enter Clinician credentials: doctor@genomeai.lab / doctor123 (or Admin: admin@genomeai.lab / admin123 ). The user session token is stored and the navigation menu reveals the Patient Directory , Variant Workspace , Case History , and Admin Dashboard . Explore the Patient Directory ( /patients ) : Access Patient Directory from the sidebar navigation. Use the real-time search bar to filter by name ( Marcus ) or condition ( Hypercholesterolemia ). Select patient Marcus Thorne ( PAT-2026-0002 , DOB: 1984-06-12, Male). Review his documented indication: Severe hypercholesterolemia, tendon xanthomas, strong family history of premature CAD . Launch the Variant Workspace ( /variants ) : Click Analyze Variant Profile on Marcus Thorne's card. The Variant Workspace opens with Marcus Thorne pre-selected and his linked demonstration panel ( demo_familial_hypercholesterolemia.vcf ) loaded in the workspace buffer. Execute Automated Pipeline : Click Run Variant Pipeline . The streaming validator checks VCF 4.2 syntax and reference coordinates (GRCh38). The annotation engine queries ClinVar and evaluates the 10-rule waterfall algorithm. Inspect Prioritized Findings : The results table renders the prioritized variant: chr19:11200000 A>G in the LDLR gene. Classification: Tier 1 (Pathogenic) | Priority Score: 95.0 | Review Flag: REVIEW_HIGH . Quality metrics confirm PASS filter status and high read depth. Consult the Contextual Genomic Assistant : Open the Genomic Assistant drawer. Type: "Explain why this LDLR mutation is classified as Tier 1 and summarize clinical implications." The assistant references the active case context, details the missense substitution disruption in LDL receptor function, and reminds the user of mandatory non-diagnostic disclaimers. Save the Case to the Ledger : Enter clinical notes in the review box: "Findings consistent with familial hypercholesterolemia; recommend cascade screening for first-degree relatives." Click Save Case to History . The system computes the SHA-256 hash of the VCF, stores the file securely in data/vcf_storage/ , creates an immutable case snapshot, and issues sequential case ID CASE-2026-0082 . View the Standalone Clinical Report ( /report/:caseId ) : Click View Report on the confirmation dialog (or navigate directly to http://localhost:5173/report/CASE-2026-0082 ). The standalone report displays clean, print-ready styling featuring patient demographics, laboratory accession info, prioritized variant tables, quality metrics, and non-diagnostic disclaimers. Verify Case History & Audit Ledger ( /cases ) : Navigate to Case History from the sidebar. Locate case CASE-2026-0082 in the auditable case registry. Click Download VCF : The backend recalculates the SHA-256 hash of the stored file on disk, validates it against the database ledger, and streams the original file to your browser. 📈 Performance Benchmark | Model Architecture | Accuracy | Precision | Recall | F1 Score | Single Inference Time | Model Size | | --- | --- | --- | --- | --- | --- | --- | --- | | 1D-CNN | 94.2% | 0.941 | 0.942 | 0.941 | ~12 ms | 1.8 MB | | Bi-LSTM | 92.8% | 0.926 | 0.928 | 0.927 | ~28 ms | 2.4 MB | | Nucleotide Transformer | 96.5% | 0.964 | 0.965 | 0.964 | ~140 ms | ~200 MB | 🧪 Testing & Quality Assurance GenomeAI maintains a comprehensive test suite of 158 automated tests covering bioinformatic parsing, prioritization rules, database transactions, API security, and end-to-end workflow resilience. Running the Test Suite # Activate Python virtual environment .\venv\Scripts\Activate.ps1 # Run the complete test suite pytest tests / - v # Run targeted end-to-end reliability & integration suite pytest tests / test_phase3_e2e_reliability.py - v # Run patient & case persistence tests pytest tests / test_patient_case_persistence.py - v # Run variant API & multipart upload tests pytest tests / test_variant_api.py - v Test Suite Coverage Breakdown Test Module Tests Scope & Assertions tests/test_phase3_e2e_reliability.py 17 End-to-end patient $\rightarrow$ VCF $\rightarrow$ prioritization $\rightarrow$ case save $\rightarrow$ standalone report lifecycle, 10 MiB upload limits, SHA-256 hash verification, concurrent case creation. tests/test_patient_case_persistence.py 15 SQLAlchemy models, patient CRUD, case snapshot immutability, foreign key integrity, SQLite/PostgreSQL schema compatibility. tests/test_chat_service.py 15 Grounded conversational AI prompts, context injection from active cases, mandatory non-diagnostic disclaimers, fallback logic. tests/test_variant_api.py 19 FastAPI endpoints, multipart file intake, reference validation, error payloads, demo panel endpoints. tests/test_variant_annotator.py 20 ClinVar matching, R1–R10 deterministic waterfall rules, priority score bounds, ACMG tier assignments. tests/test_vcf_parser.py 34 Strict VCF 4.2 syntax parsing, header extraction, chromosome normalization, indel/SNV differentiation. Baseline & Core Tests 38 DNA tokenizer rules, SHAP perturbation vectorization, PDF generation, SQLite prediction history. ⭐ Project Highlights What makes GenomeAI stand out from standard Machine Learning projects: Dual Machine Learning & Clinical LIS Platform : Bridges biological foundation models with real-world laboratory information system workflows. ACMG/AMP Compliant Waterfall Engine : Deterministic R1–R10 prioritization categorizing variants into 4 standardized clinical tiers with review flags. Cryptographically Verified File Storage : SHA-256 hashing on all stored genomic files with tamper verification upon download. Auditable Case Ledger & Standalone Reports : Unique sequential CASE-YYYY-NNNN ledger with immutable snapshots accessible via dedicated URLs. Domain-Grounded Contextual AI : AI chatbot aware of active patient indications and variant classifications with strict safety guardrails. Explainable Medical AI (SHAP) : Batched perturbation attribution identifying exact nucleotide positions driving deep learning predictions. Production Architectural Rigor : Decoupled FastAPI + React 19 architecture, JWT RBAC security, SQLAlchemy ORM with SQLite/PostgreSQL support, and Pydantic v2 validation. 🔮 Future Improvements Completed in Current Release VCF File Parser & Variant Workspace : Direct upload support for standard VCF 4.2 files with streaming validation. Automated ACMG/AMP Prioritization : Deterministic R1–R10 waterfall rule engine. Role-Based Access Control (RBAC) : JWT authentication for Admin, Clinician, and Researcher personas. Database Migration & Persistent LIS : SQLAlchemy ORM with Supabase PostgreSQL and thread-safe SQLite fallback. Clinical LIS Case Ledger & Standalone Reports : Immutable snapshot persistence, CASE-YYYY-NNNN ledger, and print-ready report views. Tamper-Evident VCF Storage : SHA-256 cryptographic hashing and verification upon retrieval. Roadmap for Future Research 3D Protein Structure Preview : Integrate AlphaFold/ESMFold predictions for mutated coding sequences. Insertion/Deletion (Indel) Deep Learning Support : Expand sequence deep learning alignment beyond fixed 201-bp windows to complex indels. Multi-GPU Distributed Training : Acceleration scripts for multi-node PyTorch transformer training. HL7 FHIR Genomic Data Export : Standardized export of variant cases matching HL7 FHIR Genomics Reporting IG specifications. Integrated Genome Browser : Embed IGV.js track visualization directly in React frontend. Expanded Disease Panel : Scale classification taxonomy from 8 to 50+ genetic conditions. Model Quantization : ONNX Runtime and TensorRT optimizations for 4x faster transformer inference. 🧩 Challenges Faced Large Genomic Assembly Management : Parsing multi-gigabyte NCBI GRCh38 FASTA files required efficient indexed random-access file streaming via pyfaidx . Streaming VCF Validation : Validating large VCF files up to 10 MiB required streaming header checks without exhausting server memory or buffering full files before validation. Explainability Scaling : SHAP perturbation loops originally ran 200 sequential single-sample inferences. Vectorizing perturbations into a single batched tensor matrix reduced scoring latency from seconds to milliseconds. Concurrent Persistence & Immutable Snapshots : Designing an immutable case snapshot architecture ensured historical clinical reports remain permanently consistent even if underlying reference databases change. Cross-Framework Integration : Seamlessly unified TensorFlow/Keras (CNN/LSTM) and PyTorch (HuggingFace Transformers) under a single FastAPI prediction engine. 🎓 Learning Outcomes Advanced bioinformatic data processing (ClinVar variant mapping, VCF 4.2 parsing & FASTA extraction). ACMG/AMP clinical variant interpretation guidelines and deterministic waterfall prioritization. Deep learning architectures for sequence data (1D-CNN, Bi-LSTM, Self-Attention Foundation Transformers). Model interpretability techniques (SHAP feature attribution & batched perturbation analysis). Laboratory Information System (LIS) design, immutable audit ledgers, and SHA-256 tamper verification. Asynchronous web service development with FastAPI, Pydantic v2, Uvicorn, and SQLAlchemy ORM. Modern frontend architecture using React 19, Vite, Recharts, and CSS modules. Dynamic vector PDF generation with ReportLab. 🤝 Contributing Contributions are warmly welcomed! Please follow these steps: Fork the Repository. Create a Feature Branch ( git checkout -b feature/AmazingFeature ). Commit your Changes ( git commit -m 'Add some AmazingFeature' ). Push to the Branch ( git push origin feature/AmazingFeature ). Open a Pull Request. Please ensure all existing test suites pass before submitting pull requests ( pytest tests/ ). 📜 License Distributed under the MIT License . See LICENSE for more information. 📬 Contact & Author Suhani Keni — Lead Developer & AI Researcher GitHub : @SuhaniKeni Project Link : https://github.com/SuhaniKeni/GenomeAI LinkedIn : (Placeholder: [Your LinkedIn Profile]) Email : (Placeholder: [ your.email@example.com ]) 🙏 Acknowledgements HuggingFace Transformers for the Nucleotide Transformer base model. NCBI & ClinVar for open-access genomic datasets. FastAPI & React teams for incredible web development frameworks. ReportLab for Python PDF generation capabilities.
2. Related Work, Literature Survey & Algorithmic Foundations
AI Foundation Models for Genomics A hub for InstaDeep's cutting-edge deep learning models and research for genomics, originating from the Nucleotide Transformer and its evolutions. 🎯 Our Focus: Advancing Genomics with AI Welcome to the InstaDeep AI for Genomics repository! This is where we feature our collection of transformer-based genomic language models and innovative downstream applications. Our work in the genomics space began with The Nucleotide Transformer , developed in collaboration with Nvidia and TUM and trained on Cambridge-1, and has expanded to include projects like the Agro Nucleotide Transformer (in collaboration with Google, trained on TPU-v4 accelerators), SegmentNT , and ChatNT . Our mission is to provide the scientific community with powerful, reproducible, and accessible tools to unlock new insights from biological sequences. This repository serves as the central place for sharing our models, inference code, pre-trained weights, and research contributions in the genomics domain, with explorations into future areas like single-cell transcriptomics. We are thrilled to open-source these works and provide the community with access to the code and pre-trained weights for our diverse set of genomics language models and segmentation models. ✨ Featured Models & Research Evolutions This section highlights the key models and research directions from our team. Each entry provides a brief overview and links to detailed documentation, publications, and resources. (Detailed code examples, setup for specific models, and in-depth figures are now located in their respective documentation pages within the ./docs folder.) 🧬 The Nucleotide Transformer v3 (NTv3) A unified multi-species foundation model that brings representation learning, functional-track prediction, genome annotation and controllable sequence generation into one framework. NTv3 uses a U-Net-like architecture with single-base tokenization to model contexts up to 1 Mb at nucleotide resolution. It is pretrained on 9 trillion base pairs from OpenGenome2, then post-trained on 16,262 functional tracks from 15 animal and plant species, and reaches state-of-the-art accuracy for functional-track prediction and genome annotation across species. It can be fine-tuned into a controllable generative model for designing enhancer sequences at a requested activity level, validated experimentally by STARR-seq. One 700M-parameter backbone is trained once and then specialised per capability, so each checkpoint inherits what the others learned about sequence rather than starting over. Checkpoint Use it for NTv3_pretrained Embeddings, masked-token prediction, and your own fine-tuning NTv3_tracks Functional signal: accessibility, histone marks, TF binding, expression, from 16 kb to 1 Mb inputs NTv3_tracks_1mb The same tracks from 1 Mb inputs only: the specialist behind the NTv3 paper's main results NTv3_annot_mammals / _plants / _insects Calling genes from sequence alone, one model per clade Annotate a genome end to end with NTv3, FASTA --> GFF3: pip install " nucleotide-transformer[gpu] " ntv3-annotate --genome /data/cow/genome.fa --model_cfg mammals --out /data/cow.gff3.gz Keywords: Multi-species Foundation Model, Long-context (1 Mb), Single-base Resolution, Functional-track Prediction, Genome Annotation, Controllable Sequence Generation, U-Net Architecture, Masked Diffusion ➡️ Model Details & Usage 🧬 Genome Annotation , annotate a genome from FASTA to GFF3 with ntv3-annotate 🤗 NTv3 collection on Hugging Face 📜 Paper NTv3 locus-analysis skill The NTv3 skill runs NTv3 inside Claude Code on a genomic locus you name. You ask in one sentence. The skill asks only for what is missing, shows one plan, and runs nothing until you approve it. The result is one self-contained report page, results/<run>/ntv3_report.html , with the predicted tracks in a genome browser. It answers five asks: Predict : the tracks you name, at a gene or an exact region, from NTv3_tracks at any window up to 1,048,576 bp, on a GPU when there is one (on CPU, 16,384 bp or more is slow and comes with a warning). The model predicts the central 37.5% of the window. A gene is centred on its transcription start site. Only the requested model heads are computed. Saliency : for each track, the gradient of its mean over the predicted span at each base, centred across A, C, G and T and kept at the reference base. Compare : one SNV, the tracks for each allele, and how far each moves (alt − ref). Annotate : a FASTA, or the 1 Mb around a Predict run, with a clade ( mammals , plants , insects ): the model ( InstaDeepAI/NTv3_annot_<clade> ), decoder ( hmm_constrained by default), 1,048,576 bp windows with the central 75% kept, and the released filters. After approval it runs ntv3-annotate 's pipeline on a GPU (about 25 GB per window) and writes a GFF3 the browser draws, next to the tracks for a locus. Explain : which model, species, window, crop, bins, and model head produced each number. Browser lanes are only the ones you name: GENCODE ( knownGene ) and ClinVar ( clinvarMain ) on human, NCBI RefSeq ( ncbiRefSeq ) on mouse, or none. Another UCSC lane, such as ENCODE cCREs, DNase clusters, phastCons, or a histone mark, is added only when you name it. Naming a tissue or an assay adds no lane. Integrated Gradients, SmoothGrad, ISM, design, knockout, and attention are not offered. Start here: ntv3_skills/README.md covers requirements, installation from zero, a first ask, the run outputs, and troubleshooting. 🧬 The Nucleotide Transformer (NT) Our foundational language models leverage DNA sequences from over 3,200 diverse human genomes and 850 genomes from a wide range of species. These models provide extremely accurate molecular phenotype prediction compared to existing methods. This family includes multiple variants (e.g., 500M_human_ref, 2B5_1000G, NT-v2 series) which are detailed further in the specific documentation. Keywords: Foundational Model, Genomics, DNA/RNA, Pre-trained, Sequence Embeddings, Phenotype Prediction ➡️ Model Details, Variants & Usage 📜 Read the Paper (Nature Methods 2025) 🤗 Hugging Face Collection 🚀 Fine-tuning Notebooks (HF): ( LoRA and regular ) 🌾 Agro Nucleotide Transformer (AgroNT) A novel foundational large language model trained on reference genomes from 48 plant species, with a predominant focus on crop species. AgroNT demonstrates state-of-the-art performance across several prediction tasks ranging from regulatory features, RNA processing, and gene expression in plants. Keywords: Plant Genomics, Foundational Model, Crop Science, Gene Expression, Agriculture AI ➡️ Model Details & Usage 📜 Read the Paper (Communications Biology 2024) 🤗 Hugging Face Collection 🧩 SegmentNT (& family: SegmentEnformer, SegmentBorzoi) Segmentation models using transformer backbones (Nucleotide Transformers, Enformer, Borzoi) for predicting genomic elements at single-nucleotide resolution. SegmentNT, for instance, predicts 14 different classes of human genomic elements in sequences up to 30kb (generalizing to 50kbp) and demonstrates superior performance. Keywords: Genome Segmentation, Single-Nucleotide Resolution, Genomic Elements, U-Net, Enformer, Borzoi ➡️ Model Details & Usage (Covers SegmentNT, SegmentEnformer, SegmentBorzoi) 📜 Read the Paper (Nature Methods 2025) 🤗 Hugging Face Collection 🚀 SegmentNT Inference Notebook (HF) 💬 ChatNT A multimodal conversational agent designed with a deep understanding of DNA biological sequences, enabling interactive exploration and analysis of genomic data through natural language. Keywords: Conversational AI, Multimodal, DNA Analysis, Genomics Chatbot, Interactive Biology ➡️ Model Details & Usage 📜 Read the Paper (Nature Machine Intelligence 2025) 🤗 ChatNT on Hugging Face 🚀 ChatNT Inference Notebook (Jax) 3️⃣ Codon-NT (Exploring 3-mer Tokenization) A Nucleotide Transformer model variant trained on 3-mers (codons). This work investigates alternative tokenization strategies for genomic language models and their impact on downstream performance and interpretability. Keywords: Genomics, Language Model, Codon, Tokenization, 3-mers, Nucleotide Transformer Variant ➡️ Model Details & Usage 📜 Read the Paper (Bioinformatics 2024) 🤗 Hugging Face Link 🧬 Isoformer A model designed for learning isoform-aware embeddings directly from RNA-seq data, enabling a deeper understanding of transcript-specific expression and regulation. Keywords: RNA-seq, Transcriptomics, Isoforms, Gene Expression, Embeddings ➡️ Model Details & Usage 📜 Read the Paper (NeurIPS 2024) 🤗 Hugging Face Link 🔬 sCT (single-Cell Transformer) Our foundational transformer model for single-cell and spatial transcriptomics data. sCT aims to learn rich representations from complex, high-dimensional single-cell datasets to improve various downstream analytical tasks. Keywords: Single-cell RNA-seq, Spatial Transcriptomics, Foundational Model, Transformer, Gene Expression ➡️ Model Details & Usage 📜 Read the Paper (OpenReview preprint) 🤗 Hugging Face Link 🚀 sCT Inference Notebook (HF) 🧪 BulkRNABert BulkRNABert is a transformer-based, encoder-only foundation model designed for bulk RNA-seq data. It learns biologically meaningful representations from large-scale transcriptomic profiles. Keywords: Bulk RNA-seq, Foundational Model, Transformer, Cancer prognosis ➡️ Model Details & Usage 📜 Read the Paper (Machine Learning for Health 2024) 🤗 Hugging Face Link 🚀 BulkRNABert Inference Notebook (HF) 🔗 MOJO (Multi-Omics JOint representation) MOJO is a multimodal model designed learn embeddings of multi-omics data. It integrates bulk RNA-seq and DNA methylation data to generate powerful joint representations tailored for cancer-type classification and survival analysis. Keywords: Bulk RNA-seq, DNA Methylation, Foundational Model, Transformer, Multimodal, Cancer prognosis ➡️ Model Details & Usage 📜 Read the Paper (ICML Workshop on Generative AI and Biology 2025) 🤗 Hugging Face Link 🚀 MOJO Inference Notebook (HF) 💡 Why Choose InstaDeep's Genomic Models? Built on Strong Foundations: Leveraging large-scale pre-training and diverse genomic datasets. Cutting-Edge Research: Incorporating the latest advancements in deep learning for biological sequence analysis. High Performance: Designed and validated to achieve state-of-the-art results on challenging genomic tasks. Open and Accessible: We provide pre-trained weights, usage examples, and aim for easy integration into research workflows. Collaborative Spirit: Developed with leading academic and industry partners. Focused Expertise: Created by a dedicated team specializing in AI for genomics at InstaDeep. 🚀 Getting Started To begin using models from this repository: Clone the repository: git clone https://github.com/instadeepai/nucleotide-transformer.git cd nucleotide-transformer Set up your environment (virtual environment recommended): python -m venv .venv source .venv/bin/activate # On Windows use `source .venv\Scripts\activate` Install the package and dependencies: pip install . # Installs the local package # Or, for a general requirements file if you have one: # pip install -r requirements.txt For detailed instructions on individual models, including specific dependencies, downloading pre-trained weights, and Python usage examples, please refer to their dedicated documentation pages linked in the "Featured Models & Research Evolutions" section above (e.g., ./docs/nucleotide_transformer.md ). 🤝 Community & Support Questions & Bug Reports: Please use the GitHub Issues page. Discussions: For broader discussions or questions, please use the GitHub Discussions tab (if enabled). Stay Updated: Follow InstaDeep's official channels for announcements on new model releases and research updates.
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
ML + Genomics Playground 🧬🤖 A Quick Introduction to Machine Learning for Genomics This repository provides a hands-on, beginner-friendly tutorial that bridges the gap between machine learning and genomics . Whether you're a biologist wanting to learn computational methods or a data scientist interested in biological applications, this tutorial will guide you through the entire pipeline from raw DNA sequences to advanced AI models. 🎯 What Makes This Special 📚 Educational Focus : Every notebook is extensively commented with explanations of both biological concepts and computational methods 🔄 Progressive Learning : Start with basic sequence analysis and build up to transformer models and RAG systems 🧬 Real Biology : Work with actual genomic problems like promoter prediction and sequence classification 🛠️ Modern Stack : Learn industry-standard tools including scikit-learn, PyTorch, HuggingFace, and LangChain 💻 Self-Contained : All examples run on provided sample data with optional real dataset integration 🎓 Perfect For Bioinformatics students learning computational approaches Data scientists exploring genomics applications Researchers needing practical ML skills for biological data Anyone curious about the intersection of AI and biology 📋 What's Inside 🧬 Core Notebooks (Follow in Order): 01. Foundation: NumPy & Pandas with BioPython ( 01_numpy_pandas.ipynb ) Parse FASTA files and extract sequence features Calculate genomic statistics (GC content, length distributions) Build clean datasets for machine learning Learn : Basic bioinformatics data handling and Python scientific computing 02. Classical ML: k-mer Classification ( 02_sklearn_kmer_classification.ipynb ) Convert DNA sequences to numerical features using k-mer counting Build and evaluate promoter vs. non-promoter classifiers Understand feature engineering for biological sequences Learn : Scikit-learn, classification metrics, and sequence motif concepts 03. Deep Learning: 1D CNN with PyTorch ( 03_pytorch_cnn_dna.ipynb ) One-hot encode nucleotides for neural networks Design and train convolutional neural networks for sequence classification Understand how CNNs detect local patterns in DNA Learn : PyTorch, neural network architectures, and deep learning for genomics 04. Advanced AI: DNA Transformers ( 04_huggingface_dnabert.ipynb ) Fine-tune pre-trained DNA language models (DNABERT-style) Leverage transfer learning for genomic tasks Work with HuggingFace ecosystem for biology Learn : Transformer models, transfer learning, and modern NLP techniques for DNA 05. Applied AI: Local RAG System ( 05_langchain_local_rag.ipynb ) Build a retrieval-augmented generation system for bioinformatics notes Create local knowledge bases without external APIs Combine vector search with language models Learn : LangChain, RAG architectures, and practical AI workflows 📁 Supporting Materials: data/ – Curated mini-datasets (100 promoter/non-promoter sequences) and utility scripts data/download_public_data.py – Scripts to fetch real public genomics datasets requirements.txt – All dependencies with version compatibility 🔬 Key Learning Outcomes By completing this tutorial, you'll understand: Biological Concepts: FASTA file formats and sequence databases Promoter regions and gene regulation k-mers, motifs, and sequence patterns GC content and genomic features Computational Skills: Sequence parsing with BioPython Feature engineering for biological data Classical machine learning with scikit-learn Deep learning with PyTorch Modern NLP/transformers for genomics Building practical AI applications Professional Tools: Jupyter notebooks for reproducible research Version control and package management Data visualization and model evaluation Working with HuggingFace model hub Local AI deployment strategies 🎯 Real-world ready : All techniques scale to production genomics workflows and research applications. 🚀 Quick Start Guide Prerequisites Python 3.10+ (conda/mamba recommended) 8GB+ RAM (for transformer models) GPU optional but recommended for deep learning sections Installation # 1. Clone this repository git clone < repository-url > cd ml-genomics-tutorial # 2. Create isolated environment (recommended) conda create -n mlgenomics python=3.10 -y conda activate mlgenomics # 3. Install all dependencies pip install -r requirements.txt # 4. Launch Jupyter Lab jupyter lab # 5. Open notebooks/ and start with 01_numpy_pandas.ipynb 🎯 Learning Path 📚 For Beginners : Follow notebooks 01→02→03 in order 🚀 Experienced : Jump to any notebook based on your interest 🔬 Researchers : Focus on notebooks 04-05 for advanced techniques ⚡ Quick Demo # Test your setup with the first notebook jupyter lab notebooks/01_numpy_pandas.ipynb 🔧 Technical Details System Requirements macOS/Linux/Windows with Python 3.10+ Memory : 4GB minimum, 8GB+ recommended for transformers Storage : ~2GB for dependencies + models GPU : Optional but speeds up PyTorch/transformer training Dependencies Included Core Scientific Stack: numpy, pandas, matplotlib, scikit-learn Bioinformatics: biopython, scikit-bio Deep Learning: torch, torchvision, transformers, datasets Modern AI: langchain, huggingface-hub, faiss-cpu Development: jupyterlab, ipywidgets Compatibility Notes ✅ NumPy < 2.0 pinned for PyTorch compatibility ✅ Latest LangChain with updated imports ✅ Cross-platform file handling ⚠️ GPU acceleration requires CUDA on Linux/Windows 📊 Datasets & Extensions 🧬 Built-in Sample Data 100 DNA sequences (50 promoters + 50 non-promoters) Realistic synthetic data for immediate experimentation No downloads required – everything works offline 🌐 Optional Public Datasets The data/download_public_data.py script provides access to real genomics data: NCBI SRA / ENA : Large-scale sequencing datasets 1000 Genomes Project : Human genetic variation data ENCODE : Regulatory element annotations 10x Genomics : Single-cell RNA-seq demonstrations Model organisms : E. coli, yeast, and other reference genomes 🎯 Extension Ideas Multi-class classification : Distinguish between promoter types Sequence generation : Train models to create realistic DNA Variant calling : Detect mutations in sequencing data Gene expression : Predict expression from promoter sequences Phylogenetics : Build evolutionary trees from sequences 🤝 Contributing & Community This is an educational resource designed to grow with the community: 🐛 Found a bug? Open an issue with details 💡 Have ideas? Suggest new notebooks or improvements 📚 Want to contribute? Add new datasets or algorithms 🎓 Using in class? Share your adaptations Acknowledgments Built with gratitude for the open-source bioinformatics community and inspired by the need for accessible AI education in biology. 📄 License MIT License - Feel free to use, modify, and distribute for educational and research purposes.
4. Neural Network Backbone, Layer Specifications & Loss Formulations
🧬 AI Genomics Lab AI-powered bioinformatics research platform for genomic analysis and disease detection. Tech Stack 📋 Description AI Genomics Lab is a local-first platform for clinical genomic analysis that combines bioinformatics pipelines, a biomedical knowledge graph, and AI-assisted interpretation. The system analyzes patient DNA samples, detects genomic variants, links them to known diseases through a graph database, and generates research-grade clinical reports using LLM agents. Research use only — not for clinical diagnostics. 🧬 AI Genomics Research Platform Bioinformatics system powered by AI to detect genetic diseases from a patient's DNA using: LLMs Graph Database Deep learning models for sequences Scientific agents Bioinformatics pipelines 🎯 Project Status Status: Phase 6 COMPLETED — Phase 7 IN PROGRESS Phase Description Status Phase 1 Docker Infrastructure ✅ Phase 2 Bioinformatics Pipeline ✅ Phase 3 Data Layer (PostgreSQL + Neo4j + MinIO) ✅ Phase 4 API Layer (50+ endpoints) ✅ Phase 5 AI Integration (LLM + Agents) ✅ Phase 6 Frontend (Next.js + IGV + Cytoscape) ✅ Phase 7 Clinical Validation (GIAB benchmarks) 🔄 Phase 8 Knowledge Ingestion (ClinVar, gnomAD) 📋 Phase 9 Production Hardening 📋 Phase 10 Advanced AI 📋 🚀 Features Secure Authentication : JWT-based authentication with Argon2 password hashing, role-based access control (admin, analyst, researcher, viewer) Genome Indexing Pipeline : Nextflow-based genome indexing pipeline creating .fai, .gzi, and .sti indexes for alignment Reference Genome Management : Web interface to manage genome references with URL download and MinIO synchronization Bioinformatics Pipeline : FASTQ → BAM → VCF with BWA, SAMtools, bcftools, and GATK Knowledge Graph : Neo4j with Gene, Mutation, Disease, Protein, Drug, and Paper nodes LLM Integration : DeepSeek / MIMO API for mutation explanation and report generation AI Agents : Multi-agent system (VariantAgent, GraphAgent, LiteratureAgent, ReportAgent) Modern UI : Next.js dashboard with multiple sections (Alignment, Storage, Analysis, Samples, Reference Genomes) Settings Management : Comprehensive platform configuration with permission-based access control MinIO Storage Integration : Object storage for genome files with sync capabilities between local and cloud storage Genome Sync Service : Automated synchronization of genome files between local storage and MinIO buckets Real-time Job Monitoring : Live streaming of Nextflow pipeline logs with stage tracking and progress updates 🖼️ Screenshots 🏗️ Architecture graph TB subgraph Input["📥 Data Input"] FASTQ[FASTQ Files] FASTA[FASTA Files] BAM[BAM Files] GENOME_URL[Genome URLs] end subgraph Indexing["🧬 Genome Indexing Pipeline"] NXF[Nextflow Runner] DOWNLOAD[Download Genome] FAI[Create .fai Index] GZI[Create .gzi Index] STI[Create .sti Index] UPLOAD[Upload to MinIO] end subgraph Pipeline["🧬 Bioinformatics Pipeline"] QC[Quality Control] ALIGN[Alignment BWA-MEM] SORT[Sorting SAMtools] VAR[Variant Calling bcftools] VCF[VCF Output] end subgraph Storage["💾 Storage"] MINIO[MinIO Object Store] POSTGRES[PostgreSQL] NEO4J[Neo4j Graph DB] end subgraph AI["🤖 AI Layer"] VA[Variant Agent] GA[Graph Agent] LA[Literature Agent] RA[Report Agent] LLM[LLM OpenRouter] end subgraph API["⚡ API Layer"] FASTAPI[FastAPI Backend] end subgraph UI["🎨 User Interface"] NEXT[Next.js Frontend] ALIGN_UI[Align Genome] STORAGE_UI[Storage Manager] REF_UI[Reference Genomes] ANALYSIS_UI[Analysis Dashboard] GRAPH[Cytoscape.js Graph] BROWSER[IGV Genome Browser] end GENOME_URL --> NXF NXF --> DOWNLOAD DOWNLOAD --> FAI FAI --> GZI GZI --> STI STI --> UPLOAD UPLOAD --> MINIO FASTQ --> QC FASTA --> QC BAM --> QC QC --> ALIGN ALIGN --> SORT SORT --> VAR VAR --> VCF VCF --> NEO4J VCF --> POSTGRES FASTQ --> MINIO FASTA --> MINIO BAM --> MINIO MINIO --> ALIGN NEO4J --> GA GA --> LLM VCF --> VA VA --> LLM LLM --> RA FASTAPI --> POSTGRES FASTAPI --> NEO4J FASTAPI --> MINIO FASTAPI --> LLM FASTAPI --> NXF NEXT --> FASTAPI ALIGN_UI --> FASTAPI STORAGE_UI --> MINIO REF_UI --> FASTAPI ANALYSIS_UI --> FASTAPI GRAPH --> NEO4J Loading 🔄 Data Flow sequenceDiagram participant User participant Frontend participant API participant Nextflow participant MinIO participant Pipeline participant Neo4j participant LLM Note over User,LLM: Genome Indexing Workflow User->>Frontend: Add Genome Reference Frontend->>API: POST /api/settings/genome-references API->>Frontend: Reference saved User->>Frontend: Index Genome Frontend->>API: POST /genome/index API->>Nextflow: Execute Nextflow pipeline Nextflow->>Nextflow: Download genome Nextflow->>Nextflow: Create indexes (.fai, .gzi, .sti) Nextflow->>MinIO: Upload indexes Nextflow->>API: Streaming logs API->>Frontend: Real-time updates Nextflow->>API: Job completion API->>Frontend: Indexing complete Note over User,LLM: Storage Management User->>Frontend: Sync Genomes Frontend->>API: POST /storage/sync/genomes API->>MinIO: List genomes MinIO->>API: Genome list API->>Frontend: Sync status Note over User,LLM: Analysis Pipeline User->>Frontend: Upload Genome File Frontend->>API: POST /analysis/upload API->>MinIO: Store file MinIO->>API: File stored User->>Frontend: Run Analysis Frontend->>API: POST /analysis/run API->>Pipeline: Execute pipeline Pipeline->>API: VCF results API->>Neo4j: Store variants User->>Frontend: View Graph Frontend->>API: GET /graph/genes/{gene} API->>Neo4j: Query graph Neo4j->>Frontend: Graph data User->>Frontend: AI Analysis Frontend->>API: POST /agents/analyze API->>Neo4j: Get context API->>LLM: Explain mutation LLM->>Frontend: Analysis result User->>Frontend: Generate Report Frontend->>API: POST /agents/report API->>LLM: Generate report LLM->>Frontend: Scientific report Loading 📁 Project Structure AI-Genomics-Lab/ ├── api/ # FastAPI backend │ ├── main.py # App entry point + top-level endpoints │ ├── v1/ # Versioned API routes │ │ ├── patients.py # Patient CRUD │ │ ├── cases.py # Clinical case management │ │ ├── samples.py # Sample management + FASTQ upload │ │ ├── variants.py # Variant querying with filters │ │ ├── reports.py # Clinical report CRUD │ │ ├── pipeline_runs.py # Pipeline execution management │ │ ├── hospitals.py # Hospital registry │ │ └── clinical_catalogs.py # Cancer types, stages, histology │ ├── dependencies.py # Auth dependencies │ ├── requirements.txt # Python dependencies │ └── Dockerfile # API container ├── services/ # Core business logic │ ├── auth_service.py # JWT + Argon2 + RBAC + audit │ ├── database_service.py # PostgreSQL schema + CRUD (28 tables) │ ├── minio_service.py # MinIO object storage client │ ├── neo4j_service.py # Neo4j graph operations │ ├── llm_client.py # LLM client (DeepSeek/MIMO/OpenRouter) + cache │ ├── bio_pipeline_client.py # VCF parsing + pipeline integration │ ├── nextflow_runner.py # Nextflow execution via Docker + SSE │ ├── cache_service.py # LLM response caching │ └── genome_sync_service.py # Genome file synchronization ├── agents/ # AI agent system │ └── __init__.py # VariantAgent, GraphAgent, LiteratureAgent, ReportAgent ├── core/ # Shared FastAPI dependencies (RBAC) │ └── deps.py ├── bio-pipeline/ # Bioinformatics pipeline │ ├── Dockerfile # Ubuntu 22.04 + strobealign + SAMtools + bcftools + GATK + Nextflow │ ├── genome_index_correct.nf # Nextflow: genome download + index creation │ ├── pipeline_1_genome_prep.nf # Nextflow: genome preparation │ ├── scripts/ │ │ ├── pipeline.sh # Main analysis: FASTQ → CRAM → VCF │ │ ├── index_genome.sh # Reference genome indexing │ │ └── vcf_dashboard.sh # VCF statistics generator │ └── archive/ # Historical pipeline variants ├── graph/ # Neo4j knowledge graph │ └── schema.cypher # Schema + seed data (genes, mutations, diseases, drugs, pathways) ├── frontend/ # Next.js 16 frontend │ ├── src/ │ │ ├── app/ # Pages (dashboard, login, settings, cases) │ │ ├── components/ # React components │ │ │ ├── sections/ # Page sections (AlignGenome, Storage, Analysis, etc.) │ │ │ ├── cases/ # Clinical case wizard (10 steps) │ │ │ ├── ui/ # Design system (Button, Card, Input, Select, etc.) │ │ │ ├── GraphView.tsx # Cytoscape.js knowledge graph │ │ │ ├── VariantTable.tsx # Variant table with filters │ │ │ └── GenomeBrowser.tsx # IGV.js genome browser │ │ ├── stores/ # Zustand state management │ │ ├── hooks/ # Custom React hooks │ │ ├── types/ # TypeScript type definitions │ │ └── design-system/ # Tokens, variants, utilities │ ├── package.json │ └── Dockerfile ├── docker/ # Infrastructure │ ├── docker-compose.yml # Multi-service orchestration (7 services) │ └── dev.sh # Development helper script ├── scripts/ # Utility scripts │ └── init_database.py # Database initialization ├── datasets/ # Data directory (gitignored) │ ├── fastq/ # Patient FASTQ files │ ├── bam/ # Aligned BAM/CRAM files │ ├── vcf/ # Variant call files │ ├── reference_genome/ # Reference genome (hg38) │ ├── annotations/ # ClinVar annotation files │ └── logs/ # Pipeline execution logs ├── PLAN.md # Master project plan ├── ARCHITECTURE.md # Technical architecture details ├── CONTRIBUTING.md # Contribution guidelines ├── README.md # This file └── LICENSE # MIT License 🛠️ Tech Stack Category Technology Backend FastAPI (Python 3.11+) Database PostgreSQL 15, Neo4j 5.14 Storage MinIO (S3-compatible) AI/LLM DeepSeek, MIMO (multi-provider) Frontend Next.js 16, React 18, Tailwind CSS Visualization Cytoscape.js, IGV.js, Recharts Pipeline Orchestration Nextflow DSL2 Bioinformatics strobealign, SAMtools, bcftools, GATK Authentication JWT + Argon2 + RBAC State Management Zustand 🌐 Services and Ports Service Port Description Frontend 3000 Next.js UI API 8000 FastAPI backend Neo4j Browser 7474 Graph database UI Neo4j Bolt 7687 Graph database protocol PostgreSQL 5432 Relational database MinIO API 9000 Object storage MinIO Console 9001 Storage management UI pgAdmin 5050 Database admin UI 📊 Data in Neo4j Loaded Nodes Type Count Examples Genes 6 BRCA1, BRCA2, TP53, EGFR, KRAS, PIK3CA Mutations 6 c.68_69delAG, c.5266dupC, R273H, L858R, G12D, E545K Diseases 5 Hereditary Breast Cancer, Ovarian Cancer, Li-Fraumeni, NSCLC, Colorectal Drugs 3 Olaparib, Osimertinib, Sotorasib Pathways 4 Cell Cycle, Apoptosis, RAS-MAPK, PI3K-AKT Relationships (Gene)-[:HAS_MUTATION]->(Mutation) (Mutation)-[:CAUSES]->(Disease) (Gene)-[:INTERACTS_WITH]->(Gene) (Drug)-[:TARGETS]->(Gene) (Gene)-[:PARTICIPATES_IN]->(Pathway) 🎨 Frontend Components Dashboard Sections AlignGenomeSection Genome alignment and indexing interface: Reference genome selection with indexed status badges Read length configuration for alignment Real-time Nextflow pipeline execution with live log streaming Stage tracking (downloading, indexing, uploading) Cancel indexing and delete index functionality StorageSection MinIO storage management: List genomes available in MinIO buckets Sync genomes between local storage and MinIO Download genomes from MinIO to local storage Visual status indicators for sync progress AnalysisSection Analysis dashboard with pipeline controls: File upload for FASTQ/BAM/VCF files Pipeline execution controls Analysis status monitoring ReferenceGenomesSection Genome reference management: Add/edit/delete genome references with URL, species, build Test genome URL connectivity Manage active/inactive references SamplesSection Sample management interface: List and manage genomic samples Sample metadata editing Visualization Components GraphView Interactive knowledge graph visualization using Cytoscape.js: Nodes: Genes (blue), Mutations (red), Diseases (green) Relationships: HAS_MUTATION, CAUSES, INTERACTS_WITH Interactive: click to select, zoom, pan VariantTable Variant table with: Search by gene or position Filters by type (SNP, Indel, Structural) Pathogenicity classification (pathogenic, likely_pathogenic, uncertain, likely_benign, benign) Data export GenomeBrowser IGV.js integration: Chromosomal locus navigation Quick navigation: BRCA1, TP53, EGFR, KRAS hg38 support 📡 API Endpoints Health GET / - API information GET /health - Health status Authentication POST /api/auth/login - User login with JWT token generation POST /api/auth/logout - User logout and session cleanup GET /api/auth/me - Get current user information POST /api/auth/refresh - Refresh access token Settings (Authenticated) GET /api/settings/genome-references - Get genome references (admin only) POST /api/settings/genome-references - Create genome reference (admin only) PUT /api/settings/genome-references/{ref_id} - Update genome reference (admin only) DELETE /api/settings/genome-references/{ref_id} - Delete genome reference (admin only) POST /api/settings/genome-references/{ref_id}/test - Test genome reference URL (admin only) GET /api/settings/pipeline - Get pipeline settings (admin only) PUT /api/settings/pipeline/{key} - Update pipeline setting (admin only) GET /api/settings/ai-providers - Get AI provider configurations POST /api/settings/ai-providers/test - Test AI provider connection (admin only) GET /api/settings/ui-preferences - Get user UI preferences PUT /api/settings/ui-preferences - Update user UI preferences GET /api/settings/audit-logs - View audit logs (admin only) GET /api/settings/system-health - Get system health status Genome (Authenticated) GET /genome/indexed - Get indexing status for all genomes GET /genome/status/{genome_id} - Get indexing status for a specific genome POST /genome/index - Start genome indexing using Nextflow pipeline DELETE /genome/index/{genome_id} - Delete genome index files GET /genome/jobs - Get all genome indexing jobs GET /genome/job/{job_id} - Get genome indexing job status Storage (Authenticated) GET /storage/genomes - List genomes from MinIO storage POST /storage/sync/genomes - Sync local genomes to MinIO GET /storage/genomes/{genome_name}/status - Get sync status for a genome POST /storage/genomes/{genome_name}/download - Download genome from MinIO to local storage GET /storage/test - Test storage connectivity Analysis POST /analysis/upload - Upload genome file POST /analysis/run - Run pipeline GET /analysis/status - Pipeline status Graph GET /graph/genes/{gene} - Gene information GET /graph/mutations/{mutation} - Mutation information GET /graph/diseases/{disease} - Disease information GET /graph/search - Search graph GET /graph/statistics - Graph statistics Agents POST /agents/analyze - Analyze variant POST /agents/report - Generate report POST /agents/complete-analysis - Complete analysis LLM POST /llm/explain - Explain mutation POST /llm/generate - Text generation 🚦 Getting Started Prerequisites Docker & Docker Compose Python 3.11+ Node.js 20+ Installation Clone the repository: git clone https://github.com/rendergraf/AI-Genomics-Lab.git cd AI-Genomics-Lab Configure environment: cp .env.example .env # Edit .env with your API keys Start services: cd docker docker-compose up -d Wait for services to be ready (about 30 seconds) Verify services are running: docker ps curl http://localhost:8000/health Accessing Services Service URL Credentials Frontend http://localhost:3000 admin@company.com / admin123 API Docs (Swagger) http://localhost:8000/docs (Authentication required for protected endpoints) Neo4j Browser http://localhost:7474 neo4j / genomics MinIO Console http://localhost:9001 genomics / genomics PostgreSQL localhost:5432 genomics / genomics / genomics Quick Test # Check API health curl http://localhost:8000/health # Response: {"status":"healthy","api":"ok","database":"ok","graph":"ok","storage":"ok"} # Test authentication curl -X POST http://localhost:8000/api/auth/login \ -H " Content-Type: application/json " \ -d ' {"email":"admin@company.com","password":"admin123","remember_me":false} ' # Response: {"access_token":"eyJhbGciOiJ...","refresh_token":"...","token_type":"bearer","expires_in":1800} # Test authenticated endpoint (using the token from above) TOKEN= " your_access_token_here " curl -H " Authorization: Bearer $TOKEN " http://localhost:8000/api/auth/me # Response: {"id":1,"email":"admin@company.com","name":"Administrator","is_active":true,"roles":["admin"]} # Check available samples curl http://localhost:8000/analysis/status Stopping Services cd docker docker-compose down Pipeline Data Place your genome files in the appropriate directories: # FASTQ files (input) mkdir -p datasets/fastq # Place .fastq or .fastq.gz files here # Reference genome (supports .fa or .fa.gz) mkdir -p datasets/reference_genome # Place Homo_sapiens.GRCh38.dna_sm.toplevel.fa.gz here # Output directories (auto-created) datasets/bam # Aligned BAM files datasets/vcf # Variant call files datasets/logs # Pipeline logs datasets/annotations # Annotation files (e.g., clinvar.vcf) Development API cd api pip install -r requirements.txt uvicorn main:app --reload Frontend cd frontend npm install npm run dev 🧬 Bioinformatics Pipeline Pipeline Overview The bioinformatics pipeline processes FASTQ files through the following steps: FASTQ → strobealign → SAM (streamed) → samtools sort → BAM → CRAM → bcftools mpileup → BCF → VCF → filtered VCF → annotated VCF Tools Used Tool Purpose strobealign Sequence alignment (5-8x faster than BWA-MEM) SAMtools SAM/BAM/CRAM processing and indexing bcftools Variant calling (mpileup) and filtering GATK Genome Analysis Toolkit (optional advanced callers) htslib Low-level HTS file I/O vmtouch RAM page cache preloading for reference genomes Pipeline Features Streaming : Uses pipes to avoid writing intermediate SAM files (saves disk space) Parallel processing : Uses 4 threads for BWA and samtools Compressed reference : Supports .fa.gz - automatically decompresses on first run Smart indexing : Only reindexes if indices don't exist Detailed logging : Each step logs to /datasets/logs/{sample}_{tool}.log Reference Genome The pipeline supports both compressed and uncompressed reference genomes: # Place in datasets/reference_genome/ Homo_sapiens.GRCh38.dna_sm.toplevel.fa.gz # Recommended (3GB vs 60GB) # or Homo_sapiens.GRCh38.dna_sm.toplevel.fa On first run, the compressed file will be decompressed automatically. Running Pipeline # Via API curl -X POST http://localhost:8000/analysis/run -H " Content-Type: application/json " \ -d ' {"sample_id": "sample_001"} ' # Check status curl http://localhost:8000/analysis/status Pipeline Environment Variables REFERENCE_GENOME_GZ = /datasets/reference_genome/Homo_sapiens.GRCh38.dna_sm.toplevel.fa.gz REFERENCE_GENOME = /datasets/reference_genome/Homo_sapiens.GRCh38.dna_sm.toplevel.fa INPUT_DIR = /datasets/fastq OUTPUT_DIR = /datasets/bam VCF_OUTPUT_DIR = /datasets/vcf LOGS_DIR = /datasets/logs ANNOTATION_DIR = /datasets/annotations 🧬 Genome Indexing Pipeline Overview The genome indexing pipeline uses Nextflow to download reference genomes and create necessary indexes for alignment (.fai, .gzi, .sti). The pipeline runs in a Docker container and uploads results to MinIO for persistent storage. Nextflow Pipeline Input : Genome ID and optional URL Processes : Download, FASTA index (.fai), BGZIP index (.gzi), Strobealign index (.sti) Output : Index files uploaded to MinIO bucket Real-time monitoring : Live log streaming via Server-Sent Events (SSE) Index Types .fai : FASTA index for random access to sequences .gzi : BGZIP index for compressed FASTA files .sti : Strobealign index for fast read alignment Usage via API # Start genome indexing curl -X POST http://localhost:8000/genome/index \ -H " Content-Type: application/x-www-form-urlencoded " \ -d " genome_id=hg38&read_length=150 " # Check indexing status curl http://localhost:8000/genome/indexed # Stream logs for a job # (Implemented via SSE in frontend) VariantAgent Analyzes specific variants by querying the knowledge graph and generating clinical interpretations. GraphAgent Performs queries to Neo4j to retrieve information about genes, mutations, and diseases. LiteratureAgent Retrieves and analyzes relevant scientific literature for detected variants. ReportAgent Generates complete scientific reports including executive summary, methodology, variant analysis, and clinical interpretation. AnalysisOrchestrator Orchestrator that coordinates all agents for complete analysis. 📈 API Usage Example: Variant Analysis import requests # Analyze variant response = requests . post ( "http://localhost:8000/agents/analyze" , json = { "variant_id" : "R273H" } ) print ( response . json ()) # Generate report response = requests . post ( "http://localhost:8000/agents/report" , json = { "sample_id" : "sample_001" , "variants" : [ "BRCA1:c.68_69delAG" , "TP53:R273H" ] } ) print ( response . json ()) 🤖 Agent System VariantAgent Analyzes specific variants by querying the knowledge graph and generating clinical interpretations. GraphAgent Performs queries to Neo4j to retrieve information about genes, mutations, and diseases. LiteratureAgent Retrieves and analyzes relevant scientific literature for detected variants. ReportAgent Generates complete scientific reports including executive summary, methodology, variant analysis, and clinical interpretation. AnalysisOrchestrator Orchestrator that coordinates all agents for complete analysis. 📝 Environment Variables Configuration # Database DATABASE_URL = postgresql://genomics:genomics@postgres:5432/genomics # Neo4j NEO4J_URI = bolt://neo4j:7687 NEO4J_USER = neo4j NEO4J_PASSWORD = genomics # MinIO MINIO_ENDPOINT = minio:9000 MINIO_ACCESS_KEY = genomics MINIO_SECRET_KEY = genomics # LLM (DeepSeek / MIMO / OpenRouter) LLM_PROVIDER = deepseek DEEPSEEK_API_KEY = your_deepseek_api_key_here MIMO_API_KEY = your_mimo_api_key_here # Pipeline (optional) REFERENCE_GENOME = /datasets/reference_genome/Homo_sapiens.GRCh38.dna_sm.toplevel.fa REFERENCE_GENOME_GZ = /datasets/reference_genome/Homo_sapiens.GRCh38.dna_sm.toplevel.fa.gz 🔒 Security Authentication : JWT with Argon2 password hashing (GPU/ASIC-resistant) Authorization : Role-based access control (admin, analyst, researcher, viewer) Audit Logging : All authentication events and configuration changes logged Patient Data Protection : Pseudonymized external IDs, no PII in LLM calls API Security : CORS configuration, SQL injection prevention via parameterized queries Configuration : Environment variables for secrets, no hardcoded credentials in code Note : This is a research platform. It is NOT certified for clinical diagnostics (FDA/CE-IVD). All AI-generated reports must be reviewed by qualified personnel before any clinical decision. 🧪 Testing Critical modules include tests: Bioinformatics pipeline Variant parser Graph ingestion 🤝 Contributions Contributions are welcome! 📄 License MIT License - See LICENSE for details. Author: Xavier Araque Email: xavieraraque@gmail.com GitHub: https://github.com/rendergraf/AI-Genomics-Lab Version: 0.2 Location: Spain Date: June 2026 Generated by AI Genomics Lab
5. Training Protocol, Convergence Behavior & Experimental Evaluation
AI for Genomics Instructor Dr. Nikolay Oskolkov, Group Leader (PI) at LIOS, Riga, Lativia Course overview This course explores the application of modern AI architectures—Convolutional Neural Networks (CNNs), Long Short-Term Memory networks (LSTMs), and Transformers—to genomic and metagenomic data. Students will gain practical experience through hands-on coding labs and interactive notebooks, learning how to model sequence data, extract biologically meaningful features, and interpret results. Emphasis is placed on real-world applications, including prediction of genomic functional elements, sequence classification and source tracking, as well as biological sequence generation. Target audience and assumed background We assume some basic awareness of UNIX environment, as well as at least beginner level of R and / or Python programming. Learning outcomes By the end of this course, participants will be able to: Understand and implement LSTM and CNN architectures for genomic sequence data Apply attention mechanisms to improve genomic feature extraction and prediction Train simple Transformer models for sequence classification or functional element prediction Use notebooks to run and modify ML & DL workflows for genomics research Interpret model outputs and assess performance using biological context Schedule Before the course Time Activity Link ~ 1 h Recorded talk: Hub AI Talks: Life Science Meets AI (starting from 59:32) Video ~ 1 h In case needed: Recap on Unix Lab ~ 1 h Useful reading: Deep Learning for Life Sciences: Do we have Big Data in Life Sciences? Blog ~ 1 h Useful reading: Deep Learning for Life Sciences: Why Biology is sceptic toward AI Blog ~ 1 h Useful reading: Deep Learning for Life Sciences: Deep Learning on ancient DNA Blog ~ 1 h Useful reading: Deep Learning for Life Sciences: Deep Learning for Single Cell biology Blog ~ 1 h Useful reading: Deep Learning for Life Sciences: Deep Learning for data integration Blog ~ 1 h Useful reading: Deep Learning for Life Sciences: Deep Learning for clinical diagnostics Blog ~ 1 h Useful reading: Deep Learning for Life Sciences: Deep Learning on Microscopy Imaging Blog ~ 1 h Useful reading: Deep Learning for Life Sciences: Deep Learning on Human Microbiome Blog ~ 1 h Useful reading: Deep Learning for Life Sciences: Deep Learning on Neanderthal Genes Blog ~ 1 h Useful reading: Deep Learning for Life Sciences: LSTM to detect Neanderthal DNA Blog Day 1: 2 pm - 8 pm Berlin time Time Activity Link 14.00 - 15.00 Course outline and practical information: introductions Slides 15.00 - 15.15 Break 15.15 - 16.15 Introduction to machine and deep learning, applications in Life Sciences Slides 15.15 - 16.15 Bonus lecture: Introduction to machine learning, comparison with statistics Slides 16.15 - 16.30 Break 16.30 - 17.15 Basics of Artificial Neural Networks (ANNs), coding a vanilla ANN from scratch Slides 17.15 - 17.30 Break 17.30 - 18.30 Coding a vanilla ANN from scratch in R Lab 18.30 - 18.45 Break 18.45 - 20.00 Coding a vanilla ANN from scratch in Python (version1) Lab 18.45 - 20.00 Coding a vanilla ANN from scratch in Python (version 2) Lab Day 2: 2 pm - 8 pm Berlin time Time Activity Link 14.00 - 15.00 Autoencoder neural network applications for Cell Biology and Data Integration Slides 15.00 - 15.15 Break 15.15 - 16.15 Applications of Autoencoder neural network for data integration Lab 15.15 - 16.15 Bonus lab: Autoencoder for single cell dimensionality reduction Lab 16.15 - 16.30 Break 16.30 - 17.30 Convolutional Neural Networks (CNNs) applications for Microscopy Imaging Slides 17.30 - 18.00 Break 18.00 - 19.00 Object detection and instance segmentation of cell microscopy images Lab 19.00 - 19.15 Break 19.15 - 20.00 Bayesian neural networks applications to cell biology and clinical diagnostics Slides 19.15 - 20.00 Bonus lab: Introduction to Bayesian statistics Lab 19.15 - 20.00 Bonus lab: Bayesian CNNs for image recognition Lab 19.15 - 20.00 Bonus lab: Bayesian neural network applications to clinical diagnostics Lab Day 3: 2 pm - 8 pm Berlin time Time Activity Link 14.00 - 14.30 Bonus lecture: UMAP vs PCA for Population Genomics applications Slides 14.00 - 14.30 Bonus lab: PCA and MDS toy example for population genomics Lab 14.30 - 15.00 CNNs and RNNs applications for genomics and ancient genomics Slides 15.00 - 15.15 Break 15.15 - 16.00 Training CNNs to predict ancient status for nucleotide sequences Lab 15.15 - 16.00 Bonus lab: predicting regions of human genome with Neanderthal introgression Lab 15.15 - 16.00 Bonus lab: LSTM to predict Neanderthal introgression Lab 16.00 - 16.15 Break 16.15 - 17.15 CNN applications for metagenomics data and source tracking problem Slides 17.15 - 18.00 Predicting microbial composition from shotgun metagenomic data with CNNs Lab 18.00 - 18.30 Break 18.30 - 19.30 Attention mechanism and Transformer applications to genomic data (version 1) Lab 18.30 - 19.30 Attention mechanism and Transformer applications to genomic data (version 2) Lab 19.30 - 20.00 Questions and discussion
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)