Complete Technical Paper Content — Ai Fact Checking
1. Executive Abstract, Problem Formulation & Background
🕵️ FactCheckAI: Automated Fact-Checking System 🌟 Overview FactCheckAI is a cutting-edge automated fact-checking system that leverages the power of deep learning. It is adept at classifying statements as 'true' or 'false', making it a valuable tool in the battle against misinformation in news, social media, and speeches. 📂 Project Structure FactCheckAI.ipynb : The heart of the project, containing functions for data cleaning, preprocessing, and building and training the LSTM-based deep learning model. requirements.txt : A list of necessary Python libraries for the smooth operation of the project. 🔧 Installation Ensure you have all the tools to get started: pip install -r requirements.txt 🚀 Usage 📁 Place your dataset in the Dataset/ directory. 🧹 Run FactCheckAI.ipynb to perform data cleaning, preprocessing, and to train and evaluate the model. 🧠 Model Description The model is a marvel of engineering with a sequential architecture, featuring: An Embedding layer for efficient text vectorization. Bidirectional LSTM layers to capture the intricate context within the text. Dense layers adept at classification tasks. Trained on meticulously preprocessed text, the model outputs a binary classification, shedding light on the veracity of input statements. 📊 Data FactCheckAI works with JSON formatted datasets, expecting statements and corresponding truth labels: { "verdict" : " true " , "statement" : " The earth orbits the sun " } 👐 Contributing Got ideas to make FactCheckAI even better? Contributions are more than welcome! Check out the contribution guidelines for more details. 📜 License Distributed under the MIT License. See LICENSE for more information.
FactCheckAI is a cutting-edge automated fact-checking system that leverages the power of deep learning. It is adept at classifying statements as 'true' or 'false', making it a valuable tool in the battle against misinformation in news, social media, and speeches.
Trained on meticulously preprocessed text, the model outputs a binary classification, shedding light on the veracity of input statements.
FactCheckAI works with JSON formatted datasets, expecting statements and corresponding truth labels:
Got ideas to make FactCheckAI even better? Contributions are more than welcome! Check out the contribution guidelines for more details.
CapCheck AI Technical Architecture Public documentation for the CapCheck AI fact-checking platform. This repo contains architecture decisions, design patterns, and technical deep-dives without proprietary code. Overview CapCheck is a fact-checking platform that uses a LangGraph-based multi-agent pipeline to verify claims in real-time. The system combines: Claude (Sonnet for reasoning, Haiku for classification/OCR) Perplexity Sonar Pro for fact-checking (#1 on SimpleQA benchmark, 0.858 F-score) Vision Transformer for AI-generated image detection OpenAI Whisper for audio transcription Architecture Overview ┌─────────────────────────────────────────────────────────────────────────────┐ │ CapCheck V2 Verification Pipeline │ └─────────────────────────────────────────────────────────────────────────────┘ Request Response │ ▲ ▼ │ ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐ ┌──────────┐ │ INTAKE │──▶│ CLASSIFY │──▶│ STRATEGY │──▶│ SPECIALISTS │──▶│SYNTHESIS │ │ │ │ │ │ │ │ (parallel) │ │ │ │ • OCR │ │ • Satire │ │ • Agent │ │ ┌───────────┐ │ │ • Verdict│ │ • Audio │ │ • Stakes │ │ select │ │ │web_search │ │ │ • Conf. │ │ • URL │ │ • AI rel │ │ • Thresh │ │ │fact_check │ │ │ • Review │ └─────────┘ └────┬─────┘ └──────────┘ │ │ai_detect │ │ └──────────┘ │ │ │claim_extr │ │ │ satire? │ └───────────┘ │ ▼ └───────────────┘ ┌──────────┐ │ OUTPUT │ ◀─── Short-circuit for satire │ (format) │ └──────────┘ Key Technical Patterns 1. Evidence-Only Synthesis All verdicts are determined from retrieved evidence only. The LLM is explicitly constrained from using training knowledge to prevent hallucinations. → See decisions/001_EVIDENCE_ONLY_SYNTHESIS.md 2. Confidence Calibration The system detects when LLMs output default confidence values (like 60%) and recalculates based on source quality, quantity, and consensus. → See decisions/002_CONFIDENCE_CALIBRATION.md) 3. Model Selection Strategy Different models for different tasks based on latency, cost, and capability requirements. → See decisions/003_MODEL_SELECTION.md 4. Graceful Degradation Automatic fallback chain when services fail: V2 → V1, Perplexity → Brave Search + Claude. → See decisions/04_GRACEFUL_DEGRADATION.md Documentation Document Description ARCHITECTURE.md Full system architecture and data flow LLM_PATTERNS.md LLM integration patterns and prompts decisions/ Architecture Decision Records (ADRs) Tech Stack Backend: TypeScript, Node.js, Fastify, LangGraph, PostgreSQL, Hasura GraphQL AI/ML: Claude Sonnet 4, Claude Haiku 3.5, Perplexity Sonar Pro, OpenAI Whisper, Vision Transformer Frontend: iOS (Swift/SwiftUI), Web (Svelte 5/SvelteKit), Chrome Extension (planned) Infrastructure: Docker, Fly.io, Cloudflare Performance Metric Value Cost per verification $0.025-0.045 Latency (P95) 40-60s full pipeline Caching Database-backed with perceptual image hashing Why This Architecture? Fact-checking requires accuracy over speed and transparency over black-box verdicts . Key design principles: Evidence-grounded - Never trust LLM training data for factual claims Auditable - Every verdict has traceable source citations Calibrated - Confidence scores reflect actual evidence quality Resilient - Graceful degradation when services fail Contact Aaron Kantrowitz aaronkantrowitz.com | LinkedIn This is a portfolio documentation repo. The CapCheck codebase is private.
Public documentation for the CapCheck AI fact-checking platform. This repo contains architecture decisions, design patterns, and technical deep-dives without proprietary code.
CapCheck is a fact-checking platform that uses a LangGraph-based multi-agent pipeline to verify claims in real-time. The system combines:
┌─────────────────────────────────────────────────────────────────────────────┐ │ CapCheck V2 Verification Pipeline │ └─────────────────────────────────────────────────────────────────────────────┘ Request Response │ ▲ ▼ │ ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐ ┌──────────┐ │ INTAKE │──▶│ CLASSIFY │──▶│ STRATEGY │──▶│ SPECIALISTS │──▶│SYNTHESIS │ │ │ │ │ │ │ │ (parallel) │ │ │ │ • OCR │ │ • Satire │ │ • Agent │ │ ┌───────────┐ │ │ • Verdict│ │ • Audio │ │ • Stakes │ │ select │ │ │web_search │ │ │ • Conf. │ │ • URL │ │ • AI rel │ │ • Thresh │ │ │fact_check │ │ │ • Review │ └─────────┘ └────┬─────┘ └──────────┘ │ │ai_detect │ │ └──────────┘ │ │ │claim_extr │ │ │ satire? │ └───────────┘ │ ▼ └───────────────┘ ┌──────────┐ │ OUTPUT │ ◀─── Short-circuit for satire │ (format) │ └──────────┘
All verdicts are determined from retrieved evidence only. The LLM is explicitly constrained from using training knowledge to prevent hallucinations.
The system detects when LLMs output default confidence values (like 60%) and recalculates based on source quality, quantity, and consensus.
Different models for different tasks based on latency, cost, and capability requirements.
Automatic fallback chain when services fail: V2 → V1, Perplexity → Brave Search + Claude.
2. Related Work, Literature Survey & Algorithmic Foundations
AI/ML: Claude Sonnet 4, Claude Haiku 3.5, Perplexity Sonar Pro, OpenAI Whisper, Vision Transformer
Fact-checking requires accuracy over speed and transparency over black-box verdicts . Key design principles:
Public documentation for the CapCheck AI fact-checking platform. This repo contains architecture decisions, design patterns, and technical deep-dives without proprietary code.
are there open source projects that use large language models connected to Internet to recursively find evidences to backup or dispute a given claim? Essentially the projects can get the bottom of atomic evidences to verify a claim? Yes, there are several open-source projects that leverage large language models (LLMs) connected to the internet to recursively find evidence supporting or disputing given claims. These projects aim to verify claims by retrieving and analyzing relevant information from external sources. Here are some notable examples: Open-RAG Open-RAG is a framework designed to enhance reasoning capabilities in Retrieval-Augmented Generation (RAG) using open-source LLMs. It transforms a dense LLM into a parameter-efficient sparse mixture of experts (MoE) model, capable of handling complex reasoning tasks, including both single- and multi-hop queries. Open-RAG trains the model to navigate challenging distractors and integrates external knowledge effectively for more accurate and contextually relevant responses ( Papers with Code ). FactLLaMA FactLLaMA combines instruction-following language models with external evidence retrieval to enhance fact-checking performance. It leverages search engines to retrieve relevant evidence for a given claim and then fine-tunes an open-source language model using this evidence, enabling it to predict the veracity of the claim more accurately ( arXiv ). RAPTOR RAPTOR introduces a novel approach to retrieval-augmented language models by constructing a recursive tree structure from documents. This structure allows for more efficient and context-aware information retrieval across large texts, addressing common limitations in traditional language models ( GitHub ). Loki Loki is an open-source tool for fact verification that integrates LLMs with search engines or specialized databases to expand evidence retrieval. It decomposes complex claims into atomic sub-claims and verifies each sub-claim by retrieving relevant information from the web, thus providing a comprehensive assessment of the original claim ( arXiv ). These projects exemplify the use of LLMs in conjunction with internet-based evidence retrieval to verify claims by decomposing them into atomic components and systematically searching for supporting or contradicting information.
are there open source projects that use large language models connected to Internet to recursively find evidences to backup or dispute a given claim? Essentially the projects can get the bottom of atomic evidences to verify a claim?
Yes, there are several open-source projects that leverage large language models (LLMs) connected to the internet to recursively find evidence supporting or disputing given claims. These projects aim to verify claims by retrieving and analyzing relevant information from external sources. Here are some notable examples:
Open-RAG Open-RAG is a framework designed to enhance reasoning capabilities in Retrieval-Augmented Generation (RAG) using open-source LLMs. It transforms a dense LLM into a parameter-efficient sparse mixture of experts (MoE) model, capable of handling complex reasoning tasks, including both single- and multi-hop queries. Open-RAG trains the model to navigate challenging distractors and integrates external knowledge effectively for more accurate and contextually relevant responses ( Papers with Code ).
FactLLaMA FactLLaMA combines instruction-following language models with external evidence retrieval to enhance fact-checking performance. It leverages search engines to retrieve relevant evidence for a given claim and then fine-tunes an open-source language model using this evidence, enabling it to predict the veracity of the claim more accurately ( arXiv ).
RAPTOR RAPTOR introduces a novel approach to retrieval-augmented language models by constructing a recursive tree structure from documents. This structure allows for more efficient and context-aware information retrieval across large texts, addressing common limitations in traditional language models ( GitHub ).
Loki Loki is an open-source tool for fact verification that integrates LLMs with search engines or specialized databases to expand evidence retrieval. It decomposes complex claims into atomic sub-claims and verifies each sub-claim by retrieving relevant information from the web, thus providing a comprehensive assessment of the original claim ( arXiv ).
These projects exemplify the use of LLMs in conjunction with internet-based evidence retrieval to verify claims by decomposing them into atomic components and systematically searching for supporting or contradicting information.
There are indeed open-source projects that aim to use large language models (LLMs) connected to the internet to recursively find evidence to support or dispute claims. These projects are part of a broader effort to enhance fact-checking and information verification using AI. Here are some notable examples:
WebGPT is an open-source project that combines GPT-3 with web browsing capabilities[1]. While not specifically designed for recursive fact-checking, it demonstrates the potential of connecting LLMs to internet resources for information retrieval and verification.
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
Several open-source projects focus on factual question answering, which can be adapted for claim verification:
KILT (Knowledge Intensive Language Tasks) This framework includes tasks related to fact-checking and provides a unified interface for knowledge-intensive NLP tasks[1].
FEVER (Fact Extraction and VERification) An open-source dataset and system for automatic fact-checking[1].
While not yet widely implemented, the concept of recursive fact-checking using LLMs is an active area of research:
Chain-of-Thought Prompting This technique encourages LLMs to break down complex reasoning tasks, which could be applied to recursive fact-checking[1].
Tree of Thoughts An extension of chain-of-thought prompting that explores multiple reasoning paths, potentially useful for in-depth claim verification[1].
Implementing recursive fact-checking with LLMs connected to the internet faces several challenges:
While there isn't a widely adopted open-source project specifically designed for recursive fact-checking using LLMs connected to the internet, the building blocks exist in various projects. Researchers and developers are actively working on combining these elements to create more sophisticated fact-checking systems.
As the field progresses, we can expect to see more integrated solutions that leverage LLMs, internet connectivity, and recursive reasoning to provide comprehensive claim verification capabilities.
This comparison outlines the distinct features and approaches of each tool, aiding in selecting the most suitable one for specific applications.
Certainly! I'll create a table comparing the projects and approaches mentioned in the context of using large language models for evidence-finding and claim verification. Please note that some of these are not direct comparisons as they serve different purposes within the broader goal of AI-assisted fact-checking.
WebGPT : While it has internet connectivity, its recursive capabilities for fact-checking are limited and not its primary focus.
KILT and FEVER : These projects provide frameworks and datasets for fact-checking but don't inherently include internet connectivity or recursive searching. They can be extended or combined with other techniques for these purposes.
4. Neural Network Backbone, Layer Specifications & Loss Formulations
Chain-of-Thought and Tree of Thoughts : These are prompting techniques rather than full projects. They demonstrate the potential for recursive reasoning in LLMs, which could be applied to fact-checking when combined with internet connectivity.
Open Source : All of these projects or techniques have open-source components, but the underlying large language models (e.g., GPT-3) may not be open-source.
Stage of Development : This varies from established datasets and techniques to more recent research proposals.
It's important to note that a comprehensive, open-source project that fully integrates LLMs, internet connectivity, and recursive fact-checking is still an emerging area of research and development. The existing projects and techniques provide building blocks that could be combined to create such a system in the future.
FactShield AI – AI-Powered Fact-Checking System FactShield AI is a multi-agent deep research fact-checking system built with LangChain and LangGraph. It clarifies user claims, orchestrates parallel research with tools, and produces a polished, citation-rich Markdown report. Use it via a CLI workflow or a Streamlit web app. Key Features Claim clarification and scoping to ensure verifiability Structured claim brief generation Multi-agent supervisor that can delegate parallel research tasks Web search via Tavily with evidence summarization and stance tagging Deliberate “think” steps between searches for higher quality Research compression and final report generation with inline citations Outputs saved to final_reports/ and downloadable in the Streamlit UI Workflow Architecture Overview FactShield uses LangGraph state machines across three layers: Scope layer ( factchecker_agent_scope.py ) Nodes: clarify_fact_request → write_claim_statement Routes to clarification or produces a structured claim brief. Supervisor layer ( factchecker_multi_agent_supervisor.py + state_multi_agent_supervisor.py ) Supervisor decides research strategy and tool usage. Tools: think_tool (reflection), ConductResearch (delegates to sub-agents), ResearchComplete . Can launch multiple fact-checker sub-agents in parallel when beneficial. Fact-checker sub-agent ( factchecker_agent.py + state_research.py ) Loop: llm_call → (tools) → compress_research Tools: tavily_search (web search + summarize), think_tool . Final report ( main.py ) Aggregates notes from the supervisor and generates a Markdown report saved to final_reports/ . Model configuration lives in utils.py via create_llm() and create_compress_llm() . Prompts are in prompts.py . Terminal UX helpers are in niceterminalui.py . Requirements Python 3.13+ API keys (depending on your chosen LLM provider and search): Tavily: TAVILY_API_KEY Google GenAI (default): GOOGLE_API_KEY or OpenAI: OPENAI_API_KEY or Groq: GROQ_API_KEY Dependencies are declared in pyproject.toml (LangChain, LangGraph, Streamlit, Rich, Tavily, etc.). Setup Create and activate a virtual environment python -m venv .venv source .venv/bin/activate Install dependencies pip install -e . If you use uv (optional): uv venv source .venv/bin/activate uv pip install -e . Configure environment variables (create a .env in the repo root) # LLM provider and model (defaults shown match utils.py) LLM_PROVIDER = google_genai LLM_MODEL = gemini-2.5-flash LLM_TEMPERATURE = 0.1 # Compression model for long summaries COMPRESS_LLM_PROVIDER = google_genai COMPRESS_LLM_MODEL = gemini-2.5-flash COMPRESS_LLM_TEMPERATURE = 0.1 COMPRESS_LLM_MAX_TOKENS = 32000 # Search TAVILY_API_KEY = your_tavily_api_key # Provider-specific API keys (set the one that matches LLM_PROVIDER) GOOGLE_API_KEY = your_google_api_key OPENAI_API_KEY = your_openai_api_key GROQ_API_KEY = your_groq_api_key Run CLI (interactive): python main.py Streamlit app: python run_streamlit.py # or streamlit run streamlit_app.py The CLI saves the final report in final_reports/ as a Markdown file with a sanitized title. The Streamlit app also lets you download the report directly from the browser.
FactShield AI is a multi-agent deep research fact-checking system built with LangChain and LangGraph. It clarifies user claims, orchestrates parallel research with tools, and produces a polished, citation-rich Markdown report. Use it via a CLI workflow or a Streamlit web app.
Supervisor layer ( factchecker_multi_agent_supervisor.py + state_multi_agent_supervisor.py )
Model configuration lives in utils.py via create_llm() and create_compress_llm() . Prompts are in prompts.py . Terminal UX helpers are in niceterminalui.py .
Dependencies are declared in pyproject.toml (LangChain, LangGraph, Streamlit, Rich, Tavily, etc.).
# LLM provider and model (defaults shown match utils.py) LLM_PROVIDER = google_genai LLM_MODEL = gemini-2.5-flash LLM_TEMPERATURE = 0.1 # Compression model for long summaries COMPRESS_LLM_PROVIDER = google_genai COMPRESS_LLM_MODEL = gemini-2.5-flash COMPRESS_LLM_TEMPERATURE = 0.1 COMPRESS_LLM_MAX_TOKENS = 32000 # Search TAVILY_API_KEY = your_tavily_api_key # Provider-specific API keys (set the one that matches LLM_PROVIDER) GOOGLE_API_KEY = your_google_api_key OPENAI_API_KEY = your_openai_api_key GROQ_API_KEY = your_groq_api_key
The CLI saves the final report in final_reports/ as a Markdown file with a sanitized title. The Streamlit app also lets you download the report directly from the browser.
FailSafe: Automated Fact-Checking System FailSafe is an open-source, modular framework designed to automate the verification of textual claims. It employs a multi-stage pipeline that integrates Large Language Models (LLMs) with retrieval-augmented generation (RAG) techniques to decompose complex arguments, retrieve high-quality evidence, and evaluate factuality with traceability. Theoretical Foundation FailSafe is not just a wrapper around an LLM; it is an implementation of recent research in cognitive architectures and automated reasoning. It addresses key challenges like Sycophancy (LLMs agreeing with users) and Snowball Hallucinations . Read our full Technical Whitepaper for a deep dive into the research methodology, including Multi-Agent Debate, H-Neuron suppression, and FActScore atomic verification. System Architecture The system operates on a four-stage pipeline designed to mitigate hallucinations and ensure comprehensive coverage of the input text. graph TD Input[User Input] --> Layer0[Layer 0: Screening] Layer0 -->|Pass| Layer1[Layer 1: Decomposition] Layer0 -->|Fail| EarlyExit[Early Warning System] Layer1 --> SAG[Structrued Argumentation Graph] SAG --> Claims{Extract Claims} Claims -->|Check-worthy?| Layer2[Layer 2: Hybrid Retrieval] Layer2 --> Cache{Knowledge Base} Cache -->|Hit| Layer3 Cache -->|Miss| Search[Query Generation & Web Search] Search --> Evidence Evidence --> Layer3[Layer 3: Verification] Layer3 --> Consistency[Cross-Examining Evidence] Consistency --> Verdict[Status: Supported/Refuted] Verdict --> Layer4[Layer 4: Synthesis] Layer4 --> FinalReport[Final Report & References] Loading Core Modules 1. Pre-computation & Screening (Layer 0) Before expensive processing, the system filters content using lightweight models: Metadata Analysis : Evaluates source credibility (if URL provided). Stylometry : Detects sensationalist patterns using NLP heuristics to flag potential misinformation early. 2. Decomposition (Layer 1) Instead of treating text as a blob, FailSafe converts it into a Structured Argumentation Graph (SAG) . Components : factcheck.core.Decompose Function : Resolves coreferences and breaks complex sentences into atomic, verifiable claims, discarding subjective opinions. 3. Verification & Retrieval (Layer 2 & 3) The system uses a hybrid approach to ground its verifications: Vector Database (ChromaDB) : Stores previously verified facts for low-latency retrieval (Caching). Deep Search (Serper/Google) : Fetches real-time information for novel claims. Consensus Mechanism : Multiple evidence sources are analyzed. A claim is only marked as "Supported" if a majority of high-trust sources agree. Technical Stack Orchestration : Python 3.10+, Celery, Redis (for asynchronous task management). LLM Integration : Abstracted layer supporting Google Gemini, OpenAI GPT-4, and Anthropic Claude. Storage : ChromaDB (Vector Store), SQLite (Metadata). Frontend : Flask-based interface for interaction and visualization. Installation and Setup Docker Deployment (Recommended) The system is fully containerized for consistency across environments. # 1. Clone the repository git clone https://github.com/Amin7410/FailSafe-AI-Powered-Fact-Checking-System.git cd FailSafe-AI-Powered-Fact-Checking-System # 2. Configure Environment variables cp .env.example .env # Required: GOOGLE_API_KEY, SERPER_API_KEY # 3. Build and Start Services docker compose up -d --build Local Development Dependencies : pip install -r requirements.txt Infrastructure : Ensure a local Redis instance is running ( redis-server ). Worker : celery -A tasks worker --loglevel=INFO Application : python webapp.py Configuration Configuration is managed via config.yaml . Key parameters include: pipeline.retriever : Switch between hybrid (Search + Vector) or serper (Search only). llm.default_model : Specifies the reasoning engine (e.g., gemini-2.0-flash ). vectordb.embedding_model : Model used for semantic search (default: intfloat/e5-base-v2 ). As a Python Library FailSafe can be easily integrated into your own Python projects: from factcheck import FactCheck # Initialize the system with default configuration fc = FactCheck . from_config () # Run the pipeline result = fc . check_text_with_progress ( "The Eiffel Tower is in Rome." ) print ( f"Verdict: { result [ 'verdict' ] } " ) print ( f"Summary: { result [ 'summary' ][ 'message' ] } " ) Live Case Studies & Demos See FailSafe in action with these real-world examples: Deep Scientific Analysis (Complex) : Content: A forensic scientific report analyzing a technical text about the role of Large Language Models (LLMs) in biomedicine. Goal: Demonstrates the system's ability to verify complex scientific claims against peer-reviewed literature. Result: 89% Highly Factual. It correctly validates claims about AI advancements while identifying nuances in "minimal effort" repurposing. Visual Fact-Check Report (General) : Content: A visual fact-check report for the claim: "Drinking lemon can cure all kinds of cancers." Goal: Demonstrates the system's ability to debunk dangerous health misinformation using authoritative medical sources (e.g., National Academies, Cancer Research UK). Result: 0% Mostly False. It unequivocally refutes the existence of a "universal cure" and identifies logical fallacies. Visual Fact-Check Report (General) : Content: A multi-claim "fake news" scenario featuring: Future events (e.g., "OpenAI has officially released GPT-6", "AI granted human rights"). Pseudoscience ("Alkaline Diet cures cancer"). Misattributed quotes (Albert Einstein on technology). Goal: Demonstrates the system's capacity to handle multiple disparate claims within a single narrative and identify logical impossibilities and anachronisms. Result: 0% Misinformation/Fiction. Classified as a complete fabrication. Note: These files are actual outputs generated by the FailSafe engine. Download and open the HTML files in your browser to interact with the full analytics. Contributing We welcome contributions! Please see our CONTRIBUTING.md for guidelines. Acknowledgements This project was inspired by and built upon the architectural framework of Loki . We acknowledge their pioneering work in automated fact-verification and their contribution to the open-source community. License This project is licensed under the MIT License - see the LICENSE file for details.
FailSafe is an open-source, modular framework designed to automate the verification of textual claims. It employs a multi-stage pipeline that integrates Large Language Models (LLMs) with retrieval-augmented generation (RAG) techniques to decompose complex arguments, retrieve high-quality evidence, and evaluate factuality with traceability.
5. Training Protocol, Convergence Behavior & Experimental Evaluation
FailSafe is not just a wrapper around an LLM; it is an implementation of recent research in cognitive architectures and automated reasoning. It addresses key challenges like Sycophancy (LLMs agreeing with users) and Snowball Hallucinations .
Read our full Technical Whitepaper for a deep dive into the research methodology, including Multi-Agent Debate, H-Neuron suppression, and FActScore atomic verification.
The system operates on a four-stage pipeline designed to mitigate hallucinations and ensure comprehensive coverage of the input text.
graph TD Input[User Input] --> Layer0[Layer 0: Screening] Layer0 -->|Pass| Layer1[Layer 1: Decomposition] Layer0 -->|Fail| EarlyExit[Early Warning System] Layer1 --> SAG[Structrued Argumentation Graph] SAG --> Claims{Extract Claims} Claims -->|Check-worthy?| Layer2[Layer 2: Hybrid Retrieval] Layer2 --> Cache{Knowledge Base} Cache -->|Hit| Layer3 Cache -->|Miss| Search[Query Generation & Web Search] Search --> Evidence Evidence --> Layer3[Layer 3: Verification] Layer3 --> Consistency[Cross-Examining Evidence] Consistency --> Verdict[Status: Supported/Refuted] Verdict --> Layer4[Layer 4: Synthesis] Layer4 --> FinalReport[Final Report & References]
Instead of treating text as a blob, FailSafe converts it into a Structured Argumentation Graph (SAG) .
# 1. Clone the repository git clone https://github.com/Amin7410/FailSafe-AI-Powered-Fact-Checking-System.git cd FailSafe-AI-Powered-Fact-Checking-System # 2. Configure Environment variables cp .env.example .env # Required: GOOGLE_API_KEY, SERPER_API_KEY # 3. Build and Start Services docker compose up -d --build
from factcheck import FactCheck # Initialize the system with default configuration fc = FactCheck . from_config () # Run the pipeline result = fc . check_text_with_progress ( "The Eiffel Tower is in Rome." ) print ( f"Verdict: { result [ 'verdict' ] } " ) print ( f"Summary: { result [ 'summary' ][ 'message' ] } " )
Note: These files are actual outputs generated by the FailSafe engine. Download and open the HTML files in your browser to interact with the full analytics.
This project was inspired by and built upon the architectural framework of Loki . We acknowledge their pioneering work in automated fact-verification and their contribution to the open-source community.
FailSafe: An autonomous fact-checking framework leveraging Multi-Agent LLMs and Structured Argumentation Graphs (SAG) to verify claims with deep-web retrieval and reasoning.
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)