Ai A B Testing | AI Final Year Project Report

Complete Technical Paper Content — Ai A B Testing

1. Executive Abstract, Problem Formulation & Background

AI-Powered A/B Testing Engine An intelligent, production-ready A/B testing system that leverages Amazon Bedrock for AI-driven variant selection, comprehensive behavioral tracking, and real-time personalization. Built with modern technologies and designed for enterprise-scale deployment. Overview This system revolutionizes traditional A/B testing by incorporating artificial intelligence to make intelligent variant selection decisions based on user behavior, context, and historical performance data. Unlike static random assignment, our AI engine learns from user interactions to optimize conversion rates and user experience in real-time. AWS Infrastructure Diagram Data Flow User Request Initiation Users access the A/B Testing Engine application through web browsers and mobile clients AWS WAF protects against common web exploits, SQL injection, and DDoS attacks before requests reach Amazon CloudFront Global Content Distribution Amazon CloudFront serves as the global content delivery network (CDN) Amazon CloudFront handles two distinct traffic patterns: Static Assets: HTML, CSS, JavaScript, and images are cached and served directly from the edge locations API Requests: Dynamic API calls are routed to the VPC Origin for backend processing Amazon S3 Origin (Static Content) Amazon S3 Bucket (Frontend) stores the static web application assets Amazon CloudFront retrieves and caches static content from this bucket VPC Origin (API Traffic) Amazon CloudFront VPC Origin configuration establishes a private connection to the Amazon VPC API requests are forwarded from Amazon CloudFront edge locations to the VPC Origin endpoint Traffic remains on the AWS private network without traversing the public internet Internal Load Balancing Application Load Balancer (Internal) receives API traffic from the VPC Origin ALB performs health checks on AWS Fargate tasks to ensure traffic routes only to healthy containers Distributes incoming requests across multiple Amazon ECS tasks for high availability FastAPI Processing FastAPI service receives HTTP requests from the ALB Implements the core A/B testing API endpoints AI-Powered Variant Selection FastAPI → Amazon Bedrock (via Interface Endpoint): FastAPI sends experiment context and user attributes to Amazon Bedrock Requests include MCP tool definitions for data access Amazon Bedrock uses Claude for intelligent variant selection Amazon Bedrock Guardrails: Validates input prompts for safety and compliance Filters responses to prevent sensitive data exposure Ensures AI outputs adhere to content policies FastAPI Internal Tool Execution: Amazon Bedrock requests tool execution via API responses FastAPI's BedrockMCPHandler executes tools by calling internal MCP Server methods MCP Tool Execution Amazon Bedrock AI decides which tools to call and returns tool_use instructions FastAPI's Tool Registry maps tool names to MCP Server methods via direct Python calls MCP Server executes registered tools within the same process Tool results are returned directly to Amazon Bedrock for final decision-making Data Persistence Layer Gateway Endpoint (Amazon DynamoDB) provides private connectivity to Amazon DynamoDB MCP Server performs CRUD operations on Amazon DynamoDB tables Additional Integration Flow: Container Orchestration AWS Fargate tasks run the application containers in private subnets Each task pulls container images from Amazon ECR via Interface Endpoint during launch FastAPI application contains integrated MCP Server components within the same container Monitoring & Observability Amazon CloudWatch collects logs, metrics, and traces from all components Event Logging Gateway Endpoint (Amazon S3) enables private access to Amazon S3 FastAPI and MCP write structured event logs to Amazon S3 Bucket (Event Logs) Security Architecture AWS IAM: Manages identity and access control across all services Amazon ECS Task Role for service permissions Amazon CloudFront Origin Access Identity for Amazon S3 Least-privilege policies for each component AWS KMS: Encrypts data at rest across all storage services Amazon DynamoDB table encryption with AWS KMS keys Amazon S3 bucket encryption for event logs Amazon CloudWatch log encryption Automatic key rotation policies Getting Started Quick Setup Links Frontend Setup Guide - React application with SDK integration Backend Setup Guide - FastAPI service with AI-powered variant selection Infrastructure Deployment - AWS CloudFormation templates and deployment guide SDK Documentation - TypeScript SDK for easy integration First Experiment Setup Before using the system, you need to create your first experiment. The system provides multiple ways to create experiments: Option 1: Interactive CLI (Recommended for First Time) cd backend/experiment_tools python create_experiment.py interactive Follow the prompts to create your experiment with: Experiment name and description Testing goal (conversions, engagement, etc.) Hypothesis with expected impact Variant configurations Option 2: Quick Test python create_experiment.py quick --name " My First Test " --description " Testing CTA buttons " Option 3: Using Templates # Use pre-built templates python create_experiment.py json --file experiment_templates/simple_ab_test.json Available Templates: simple_ab_test.json - Basic 2-variant test cta_optimization.json - 3-variant CTA testing seasonal_campaign.json - 4-variant seasonal campaign Experiment Configuration Details Experiments are stored in Amazon DynamoDB and include: Experiment metadata : name, description, goal, hypothesis Variant configurations : control and test variants with AI themes/tones Performance metrics : conversion rates, engagement scores, statistical significance Status management : draft, active, paused, completed For detailed experiment creation guide, see Experiment Guide . How the System Works 1. Intelligent Variant Selection For New Users: Uses hash-based assignment for balanced distribution No AI overhead, ~5ms response time Ensures statistical validity across variants For Returning Users: AI-powered selection using Amazon Bedrock Analyzes behavioral patterns, device context, session data Leverages personalization profiles and similar user patterns ~200-500ms response time with rich context analysis Fallback Strategy: Balanced assignment when AI models are unavailable Ensures system reliability and consistent user experience 2. Comprehensive Event Tracking Real-time Event Processing: Page visits, button clicks, product views, conversions Automatic scroll depth tracking (25%, 50%, 75% milestones) Cart interactions and e-commerce specific events Background processing with retry logic Behavioral Learning: Engagement scoring based on user interactions Conversion likelihood prediction User intent detection (browsing, researching, converting) Visual preference and attention span analysis 3. AI-Powered Personalization Model Context Protocol (MCP) Integration: Rich context provision to AI models User profile analysis and similar user matching Session context and behavioral signals Real-time performance data integration Personalization Features: Device-specific optimizations Behavioral pattern recognition Collaborative filtering based on similar users Dynamic content adaptation 4. Production-Ready Infrastructure Containerized Deployment: Multi-architecture Docker support (AMD64/ARM64) Amazon ECS service with auto-scaling Application Load Balancer for high availability Health checks and monitoring Data Storage: Amazon DynamoDB for experiments, events, and user profiles Amazon S3 for event logs and analytics Point-in-time recovery and backup strategies System Capabilities Core Features AI-Driven Variant Selection : Intelligent assignment based on user behavior Real-time Personalization : Dynamic content optimization Comprehensive Analytics : Performance tracking and insights Event-Driven Architecture : Scalable background processing Offline Support : Event queuing with retry logic Multi-Device Optimization : Responsive across all platforms Advanced Capabilities Behavioral Profiling : Deep user understanding and segmentation Collaborative Filtering : Similar user pattern analysis Performance Optimization : Automatic variant performance tracking Context Enrichment : Automatic request enhancement with device/session data Statistical Significance : Confidence intervals and significance testing A/B Test Management : Complete experiment lifecycle management Integration Features TypeScript SDK : Type-safe integration with comprehensive API React Context Pattern : Proper state management for React applications Event Batching : Efficient network usage with intelligent queuing Error Handling : Graceful degradation and fallback strategies Debug Mode : Comprehensive logging and monitoring capabilities Repository Structure /frontend - React TypeScript demo application with SDK integration examples /backend - FastAPI ECS service with AI-powered variant selection /sdk - Lightweight TypeScript SDK for easy integration /infrastructure - CloudFormation templates for AWS deployment Each component has detailed README files with setup instructions, API documentation, and integration examples. Security Considerations Infrastructure Security - Already Implemented Network Security: Private ALB with Amazon CloudFront VPC Origins : Internal ALB completely isolated from public internet Amazon VPC Endpoints : Private connectivity to AWS services (Amazon DynamoDB, Amazon S3, Amazon Bedrock, Amazon ECR, Amazon CloudWatch) Security Groups : Least-privilege network access with specific port/protocol restrictions Access Control & IAM: Amazon ECS Task Role : Granular Amazon DynamoDB permissions, Amazon S3 event logs access, Amazon Bedrock InvokeModel Amazon ECS Execution Role : Limited Amazon ECR and Amazon CloudWatch permissions for container operations Parameter Store Integration : Secure image tag management with proper IAM controls No Wildcard Permissions : Explicit resource ARNs for all IAM policies Data Protection: Encryption at Rest : Customer-managed AWS KMS keys for all Amazon DynamoDB tables and Amazon ECR repository Encryption in Transit : TLS 1.2+ for all communications, HTTPS enforcement via Amazon CloudFront Amazon S3 Security : Private buckets with OAC, server-side encryption, versioning enabled Application Security: AWS WAF Integration : Protection against common attacks (OWASP Top 10) Core Rule Set (CRS) for common web exploits SQL Injection Rule Set for database attack prevention Known Bad Inputs Rule Set for malicious patterns Rate limiting: 2000 requests per 5 minutes per IP Amazon CloudFront Security : HTTP/2 and HTTP/3 support, security headers policy Container Security : Amazon ECR vulnerability scanning, immutable image tags Health Monitoring : Amazon ECS health checks, Amazon CloudWatch monitoring and alerting Production Security Recommendations Critical Enhancements: SSL/TLS Certificate Management Add ACM certificate to ALB for end-to-end encryption Update security groups to require HTTPS (port 443) Configure certificate auto-renewal Enhanced WAF Configuration Add custom WAF rules for your application patterns Configure geo-blocking for high-risk regions Implement bot detection and mitigation Add custom rate limiting per API endpoint Network Security Hardening Enable VPC Flow Logs for security monitoring Configure Network ACLs for additional subnet-level protection Implement AWS Shield Advanced for DDoS protection Add CloudTrail for API activity logging Application Security: Authentication & Authorization Implement proper authentication mechanisms for production access Configure role-based access control (RBAC) for different user types Use OAuth 2.0 / OpenID Connect for secure login flows Implement API key management for programmatic access Add multi-factor authentication (MFA) for administrative users Configure session management with secure token handling Bedrock Guardrails (Optional) Amazon Bedrock Guardrails provide additional content safety and responsible AI controls for your A/B testing system. Ready-to-Use Example: Template : infrastructure/bedrock-guardrail.yaml - Complete CloudFormation template Pre-configured : Content filtering, PII protection, JSON validation, prompt injection protection Schema Validation : Pre-built schemas for variant selection and performance analysis Deployment Ready : Can be deployed directly to your AWS account Implementation Details: Location : backend/services/bedrock_service.py and backend/mcp_server/bedrock_mcp_handler.py Status : Implemented - user configurable Configuration : Environment variable controlled Quick Setup: # Deploy the provided guardrail template aws cloudformation deploy \ --template-file infrastructure/bedrock-guardrail.yaml \ --stack-name ab-testing-guardrail \ --capabilities CAPABILITY_IAM # Get the guardrail ID GUARDRAIL_ID= $( aws cloudformation describe-stacks \ --stack-name ab-testing-guardrail \ --query ' Stacks[0].Outputs[?OutputKey==`GuardrailId`].OutputValue ' \ --output text ) # Set environment variables BEDROCK_GUARDRAIL_ID= $GUARDRAIL_ID BEDROCK_GUARDRAIL_VERSION=DRAFT Pre-Configured Features in Template: Content Filtering : Blocks hate, harassment, violence, self-harm, sexual content PII Protection : Masks user IDs, emails, phone numbers, addresses JSON Enforcement : Blocks natural language responses, requires structured JSON Statistical Claims Validation : Flags unverified statistical claims for review Response Schemas : Enforces proper format for variant selection and performance analysis Benefits for A/B Testing: Enhanced content safety in AI-generated variant analysis PII protection in user behavior analysis Professional language in performance reports Compliance with content governance policies Responsible AI decision-making in variant selection Note : Guardrails add 20-50ms latency to AI calls but provide important safety controls for production deployments. This repository provides a complete, production-ready A/B testing solution that can be adapted to your specific needs, whether you're looking for a simple SDK integration or a full-scale AI-powered testing platform. Security See CONTRIBUTING for more information. License This library is licensed under the MIT-0 License. See the LICENSE file.

An intelligent, production-ready A/B testing system that leverages Amazon Bedrock for AI-driven variant selection, comprehensive behavioral tracking, and real-time personalization. Built with modern technologies and designed for enterprise-scale deployment.

This system revolutionizes traditional A/B testing by incorporating artificial intelligence to make intelligent variant selection decisions based on user behavior, context, and historical performance data. Unlike static random assignment, our AI engine learns from user interactions to optimize conversion rates and user experience in real-time.

Before using the system, you need to create your first experiment. The system provides multiple ways to create experiments:

python create_experiment.py quick --name " My First Test " --description " Testing CTA buttons "

# Use pre-built templates python create_experiment.py json --file experiment_templates/simple_ab_test.json

2. Related Work, Literature Survey & Algorithmic Foundations

Each component has detailed README files with setup instructions, API documentation, and integration examples.

Amazon Bedrock Guardrails provide additional content safety and responsible AI controls for your A/B testing system.

# Deploy the provided guardrail template aws cloudformation deploy \ --template-file infrastructure/bedrock-guardrail.yaml \ --stack-name ab-testing-guardrail \ --capabilities CAPABILITY_IAM # Get the guardrail ID GUARDRAIL_ID= $( aws cloudformation describe-stacks \ --stack-name ab-testing-guardrail \ --query ' Stacks[0].Outputs[?OutputKey==`GuardrailId`].OutputValue ' \ --output text ) # Set environment variables BEDROCK_GUARDRAIL_ID= $GUARDRAIL_ID BEDROCK_GUARDRAIL_VERSION=DRAFT

Note : Guardrails add 20-50ms latency to AI calls but provide important safety controls for production deployments.

This repository provides a complete, production-ready A/B testing solution that can be adapted to your specific needs, whether you're looking for a simple SDK integration or a full-scale AI-powered testing platform.

A/B Experimentation Platform with Intelligent Agents Modular ML Experimentation • Automated Decision-Making • Observability-First 📌 Overview This repository contains a complete A/B experimentation platform powered by agents, including automated traffic allocation, real-time metric aggregation, statistical evaluation, model inference service, retraining workflow, observability, and human-readable reporting. The system is designed as a modular, extensible, production-oriented experimentation engine that can be embedded into any product or ML workflow. 🔍 Full low-level architecture and agent specifications are provided in the included Technical Report (PDF). 📐 High-level architecture diagram is provided as a separate PDF illustration. 🚀 Key Features Event ingestion & normalization through n8n Dynamic experiment routing (A/B or multi-arm) FastAPI-based ML service: /predict /stat_test /retrain /nlq (natural-language queries) Intelligent Agents: Experiment Agent (traffic allocation) Metrics Agent (aggregation) Evaluator Agent (statistical tests) Trainer Agent (model retraining) PostgreSQL storage with audit logs, metrics tables, and model registry Observability & alerts (low-confidence predictions, pipeline issues) PDF reporting for experiment summaries Plug-and-play integration with external systems or applications 🏗 High-Level Architecture +--------------------+ | Data Sources | | (Events, Labels) | +---------+----------+ | v +-----------+-------------+ | n8n Orchestrator | | - ingestion | | - routing (A/B) | | - triggers & alerts | +-----------+-------------+ | v +-------------------+ FastAPI +------------------+ | Experiment Agent | <-------------> | Model Registry | | - traffic control | | (artifacts) | +-------------------+ +------------------+ | v +---------+----------+ | FastAPI ML API | | /predict | | /stat_test | | /retrain | +---------+----------+ | v +---------------------------------------+ | PostgreSQL | | events | labels | ab_metrics | audit | +---------------------------------------+ | v +---------+---------+ | Reporting Layer | | PDF / dashboards | +---------+---------+ | v End Users Scheme using Draw.io 📄 Documentation Structure This repository includes two primary documents: Technical Report (PDF) Full detailed architecture: agents, models, decision logic, metrics, constraints, and design rationale. look report here High-Level Architecture Diagram (PDF) A clean visualization for presentations and system overview. 🔧 Technology Stack Layer Tool Purpose Orchestration n8n Event routing, experiment workflows, automation Model Serving FastAPI Prediction, evaluation, retraining, NLQ Storage PostgreSQL Events, labels, metrics, model registry Backend Logic Python 3.10+ ML models, agents, data processing CI/CD GitHub Actions Automated build & deploy Containerization Docker Compose Local and production-ready deployments Observability (Optional) Prometheus, Grafana, Sentry Metrics & alerts 🧠 Agents Architecture (Short Summary) Each agent is fully described in the Technical Report. Below is the high-level overview: Experiment Agent Applies statistical results. Selects winning model. Adjusts A/B traffic values in real time. Writes decisions to audit log. Metrics Agent Periodically aggregates prediction events. Computes recall, precision, FPR, accuracy. Stores results in ab_metrics. Evaluator Agent Runs statistical tests via /stat_test. Generates insights (p-value, confidence interval). Creates human-readable conclusions. Trainer Agent Retrains on Kaggle or internal dataset. Applies preprocessing and versioning. Updates the Model Registry. 🔍 Observability & Alerts Low-confidence prediction alerts Model drift signals API latency & error rate monitoring Experiment health checks Full audit trail for decisions 🎛 Configuration Configuration is stored in PostgreSQL: experiment configuration traffic split ratios active model version metrics history audit logs This makes the system fully dynamic and adjustable without redeployment. 🧪 Sample User Query (NLQ) User: “Compare Model A and Model B over the last 7 days and recommend a rollout strategy.” System Response: { "winner" : " B " , "p_value" : 0.008 , "effect_size" : 0.12 , "recommendation" : " Increase B to 70% traffic; monitor FPR for 24h. " } ROADMAP Version Feature 1.0 Initial release (A/B testing, agents, metrics) 1.1 Auto-rollout strategies 1.2 PDF reports + dashboards 1.3 Multi-model (A/B/C/D) support 1.4 Canary deployments & rollback logic 2.0 Real-time drift detection 2.1 Feature store integration 3.0 Full experimentation as a service (EaaS) 🤝 Contribution Guidelines We welcome contributions! Please follow the steps below: 1. Fork the repository 2. Create a feature branch 3. Ensure code follows our style guide 4. Add or update tests 5. Submit a pull request describing your change 6. For major changes, please open an issue beforehand. 📬 Contact For questions or interest in integrating the platform into your product: 📧 boris.cherkasov@outlook.com 🌐 https://github.com/BorDch ⭐ Acknowledgements This project was built as part of an engineering challenge and later evolved into a production-capable experimentation platform.

3. Dataset Ingestion, Preprocessing & Stochastic Augmentations

This repository contains a complete A/B experimentation platform powered by agents, including automated traffic allocation, real-time metric aggregation, statistical evaluation, model inference service, retraining workflow, observability, and human-readable reporting.

The system is designed as a modular, extensible, production-oriented experimentation engine that can be embedded into any product or ML workflow.

🔍 Full low-level architecture and agent specifications are provided in the included Technical Report (PDF). 📐 High-level architecture diagram is provided as a separate PDF illustration.

Technical Report (PDF) Full detailed architecture: agents, models, decision logic, metrics, constraints, and design rationale. look report here

High-Level Architecture Diagram (PDF) A clean visualization for presentations and system overview.

4. Neural Network Backbone, Layer Specifications & Loss Formulations

Each agent is fully described in the Technical Report. Below is the high-level overview:

{ "winner" : " B " , "p_value" : 0.008 , "effect_size" : 0.12 , "recommendation" : " Increase B to 70% traffic; monitor FPR for 24h. " }

1. Fork the repository 2. Create a feature branch 3. Ensure code follows our style guide 4. Add or update tests 5. Submit a pull request describing your change 6. For major changes, please open an issue beforehand.

This project was built as part of an engineering challenge and later evolved into a production-capable experimentation platform.

The project implements a specialized AI agent in the field of A/B testing, which is able to upload data,, put forward hypotheses, check statistical tests, make predictions and return the answer to the user in the most convenient form. This project can be integrated for any ML platform and can be used to solve various tasks.

AI A/B Test Engine A universal, API-first A/B testing platform that allows you to run AI-generated experiments on any website without touching production code. This project is built using: Next.js (App Router) React, Tailwind CSS Prisma with SQLite Groq AI (Llama 3) for variant generations Overview The platform consists of three main parts: REST API & Dashboard: The core hub where tests are tracked, analytics are gathered, and projects are managed. JS SDK ( sdk.js ): A lightweight script injected into customer websites to handle deterministic variant assignment and event tracking. Admin Overlay ( overlay.js ): A visual editing interface injected directly into the customer's site. It allows you to point-and-click any element, generate AI variants, and launch a test instantly. Local Setup 1. Installation cd ai_ab_test_engine npm install 2. Environment Variables Create a .env.local file with your Groq API key: GROQ_API_KEY = gsk_... GROQ_API_URL = https://api.groq.com/openai/v1/chat/completions 3. Database Initialization npx prisma db push npx prisma generate 4. Running the Dashboard npm run dev The admin dashboard will be available at http://localhost:3000 . Integrating with a Website OCS Pages is pre-integrated ( cmnfour780000f63gu69kg2v8 in head-custom.html ). Open the dashboard to see live tests — no new project needed. For another site, expand Register another site on the dashboard and add your snippet: < script src =" https://ai-ab-test-engine.vercel.app/sdk.js " data-project-id =" YOUR_PROJECT_ID " > </ script > For Jekyll or GitHub Pages, include data-site-baseurl when assets use a subpath: < script src =" https://ai-ab-test-engine.vercel.app/sdk.js " data-project-id =" ... " data-site-baseurl =" /pages " > </ script > Create tests visually: open the site with ?ab_admin=true and use the overlay. Finalize → Pull Request (Pages repo) Configure GitHub Integration on the dashboard (PAT with repo scope, e.g. Open-Coding-Society/pages , base branch main ). When you Finalize a test with a declared winner, the engine: Resolves the Jekyll source file from the page path (stored on the Control variant when the test was launched, or sent from the overlay on finalize). Replaces the control text in that .md file with the winning variant (text, mermaid block, HTML/button, or game embed + assets/js/ab-tests/{slug}.js ). Opens a PR that edits the real page source—not a separate ab-tests/ snippet folder. Reposition tests still require a manual layout change in source.

5. Training Protocol, Convergence Behavior & Experimental Evaluation

A universal, API-first A/B testing platform that allows you to run AI-generated experiments on any website without touching production code.

< script src =" https://ai-ab-test-engine.vercel.app/sdk.js " data-project-id =" YOUR_PROJECT_ID " > </ script >

< script src =" https://ai-ab-test-engine.vercel.app/sdk.js " data-project-id =" ... " data-site-baseurl =" /pages " > </ script >

Configure GitHub Integration on the dashboard (PAT with repo scope, e.g. Open-Coding-Society/pages , base branch main ).

Recently, I was reading through A/B Testing with Machine Learning - A Step-by-Step Tutorial written by Matt Dancho of Business Science . I have been always fascinated by the idea of A/B Testing and the amount of impact it can bring in businesses. The tutorial is very definitive and Matt has explained each and every step in the tutorial. He has detailed about each and every decision taken while developing the solution. Even though the tutorial is written in R , I was able to scram through his code and my knowledge of Data Science helped me to understand the concepts very quickly. I will have to thank Matt for putting together all the key ingredients of the Data Science world and or using them to solve a real problem. The notebook in this repository contains my implementation of the solution (presented by Matt) in Python.