🎯

RAG Faithfulness & Hallucination Evaluator

A production-grade evaluation ecosystem for verifying factual grounding in LLM/RAG responses. Explicitly decouples Contextual Faithfulness from Real-World Factual Truth.

🤗 Dataset 📦 Model ⚡ Python API

Pre-loaded Adversarial Challenge Presets

Click any preset to test instantly

Input Evaluation Pair

Diagnostic Evaluation Report

Awaiting Audit
Context Grounding --% Entailment weight
Contradiction -- Direct refutation
Latency -- ms CPU execution

Atomic Claim Breakdown

0 claims parsed
Enter text or select a preset above, then click Run Audit.
Diagnostic Explanation:

Evaluates contextual entailment. Note: an ungrounded claim is not necessarily factually false in the real world; it indicates lack of evidence in the provided context.

Empirical Benchmark Leaderboard (Test Split)

Evaluated on held-out test split (38 samples, zero data leakage)

View full benchmark dataset →
Evaluator Architecture Accuracy Macro F1 Weighted F1 MAE Spearman ρ Latency (CPU)
Lexical Overlap Baseline 0.395 0.267 0.253 0.364 0.250 0.3 ms
TF-IDF Cosine Baseline 0.158 0.061 0.068 0.298 0.242 6.0 ms
Rule-Based Heuristic Guardrail 0.316 0.276 0.273 0.266 0.486 0.1 ms
LLM-as-a-Judge Simulator 0.447 0.409 0.413 0.302 0.394 0.1 ms
★ Hybrid Cross-Encoder (Flagship) 0.553 0.595 0.459 0.271 0.644 0.5 ms

Python 1-Liner Quickstart

Install the package or load the dataset directly into Python / PyTorch / Transformers:

1. Load Dataset (Hugging Face Datasets)
from datasets import load_dataset

dataset = load_dataset(
  "Kicaulah/rag-faithfulness-benchmark"
)
print(dataset["test"][0])
2. Production Evaluator Pipeline
from src.inference.pipeline import FaithfulnessPipeline

pipe = FaithfulnessPipeline()
res = pipe.evaluate(
    context=context, question=q, answer=ans
)
print("Score:", res.overall_faithfulness_score)