System Architecture Blueprint

RAG-010: Automated RAG Evaluation & Synthetic Testing Platform — Engineering Blueprint

This specification details the mathematical formulas, pipeline topologies, synthetic dataset generation engines, and CI/CD automated validation gates for RAG-010 (Automated RAG Evaluation & Synthetic Testing Platform).

1. System Specifications & Performance SLA

Evaluation Engine Throughput: $< 5.0\text{ seconds}$ per Query-Context-Response triplet evaluation utilizing local quantized LLM-as-a-judge nodes (e.g., Llama-3-8B-Instruct via vLLM).

Synthetic Dataset Generation Rate: $> 500$ high-signal QA pairs (including Ground Truth Contexts) generated per hour from raw unstructured text files.

Continuous Integration (CI) Gate SLA: 100% automated blocking of code deployments if evaluation metrics regress below specified baseline criteria (e.g., $\Delta_{\text{metric}} < -0.05$).

MLOps Visualization Latency: $< 2.0\text{ seconds}$ dashboard loading and query trace lookup using embedded telemetry data trackers (Phoenix / Arize).

1. Ingestion, Generation, & Evaluation Topology

RAG-010 operates on a dual-track loop: Dataset Synthesis (offline/ingestion track) and Performance Auditing (CI/CD / evaluation track). Rather than testing on static or manually written prompts, the system automatically builds an evolving, representative evaluation matrix from the document indices themselves.

              [ Raw Enterprise Knowledge Base (PDFs) ]
                                 │
                    (Document Chunking & Parse)
                                 ▼
                  [ Synthetic Question Generator ]
                   • Generates: Simple, Reasoning, Multi-context
                   • Generates: Ground Truth Contexts & Answers
                                 │
                                 ▼
                 [ Golden Evaluation Dataset Space ]
                                 │
                (Passed to Candidate RAG Pipeline)
                                 ▼
            [ Target Query-Context-Answer Output Triple ]
                                 │
                                 ▼
                   [ LLM-as-a-Judge Evaluator ]
                    • Evaluates: Faithfulness
                    • Evaluates: Answer Relevance
                    • Evaluates: Context Recall
                                 │
     ┌───────────────────────────┴───────────────────────────┐
     ▼ (SLA Metrics Meet Baseline)                          ▼ (Metrics Regressed)

[ CI/CD Pass / Auto-Merge ] [ Block Deploy / Trigger Alert ]

1. Mathematical Formulations of Core RAG Metrics

To replace qualitative "vibe-based" analysis, the platform scores candidates across the "RAG Triad" using deterministic semantic-extraction mathematics evaluated via structured instructions on Judge LLMs.

Let:

$Q$ represent the input user query.

$C = {c_1, c_2, \dots, c_k}$ represent the retrieved context chunks returned by the retriever.

$A$ represent the generated answer returned by the generator.

$G$ represent the gold-standard ground-truth answer.

3.1 Faithfulness (Groundedness)

Faithfulness measures whether the generated answer $A$ is derived strictly and only from the retrieved context $C$, preventing hallucinations.

Let the judge LLM extract a set of individual factual statements $S(A) = {s_1, s_2, \dots, s_N}$ from the generated answer $A$. For each statement $s_i$, the model determines a binary verification index $V(s_i, C) \in {0, 1}$, where $V(s_i, C) = 1$ if statement $s_i$ can be directly inferred from context $C$, and $0$ otherwise. The Faithfulness score $F$ is formulated as:

$$F = \frac{\sum_{i=1}^{N} V(s_i, C)}{N} \quad \text{where } N = \vert{}S(A)\vert{}$$

3.2 Answer Relevance

Answer Relevance measures whether the generated answer $A$ directly addresses the core objective of the query $Q$, penalizing redundant or off-topic outputs.

To compute relevance without relying on direct string matching, the judge LLM generates a set of $M$ synthetic queries $Q' = {q'1, q'_2, \dots, q'_M}$ based solely on the generated answer $A$. We compute the cosine similarity between the original query vector $\mathbf{e}_Q$ and the generated query vectors $\mathbf{e}{q'_i}$:

$$AR = \frac{1}{M} \sum_{i=1}^{M} \frac{\mathbf{e}Q \cdot \mathbf{e}{q'i}^T}{\Vert{}\mathbf{e}_Q\Vert{} \Vert{}\mathbf{e}{q'_i}\Vert{}}$$

3.3 Context Recall

Context Recall measures whether the retriever successfully fetched all critical elements of the ground-truth information $G$ required to formulate the response.

Let the ground-truth answer $G$ be split into a set of sentence assertions $G_{\text{sentences}} = {g_1, g_2, \dots, g_P}$. For each sentence $g_j$, we evaluate if it is present or supported within the retrieved context chunks $C$, mapping a verification index $V(g_j, C) \in {0, 1}$:

$$CR = \frac{\sum_{j=1}^{P} V(g_j, C)}{P} \quad \text{where } P = \vert{}G_{\text{sentences}}\vert{}$$

3.4 Context Precision

Context Precision measures whether the highest-relevance retrieved context chunks are ranked at the top of the context array $C$.

Let $I(k) \in {0, 1}$ be the relevance indicator of the $k$-th retrieved context chunk $c_k$ relative to the ground truth $G$. The Precision at rank $k$ is calculated as $P@k = \frac{\sum_{i=1}^{k} I(i)}{k}$. The Context Precision ($CP$) score over $K$ retrieved chunks is:

$$ CP = \frac{\sum_{k=1}^{K} \left( P@k \cdot I(k) \right)}{\sum_{k=1}^{K} I(k)} $$

1. Evaluator Engine & Dataset Synthesizer Implementation

This self-contained Python module generates a synthetic evaluation dataset from raw document strings and executes structured LLM evaluations using an isolated Judge Model interface.

import os import json import numpy as np from typing import List, Dict, Any, Tuple from pydantic import BaseModel, Field from openai import OpenAI

1. Structured Schemas for Judge Evaluation Output

class StatementExtraction(BaseModel): statements: List[str] = Field(description="List of individual factual statements extracted from the answer.")

class StatementVerification(BaseModel): statement: str = Field(description="The statement being verified.") verdict: int = Field(description="1 if the statement is fully supported by the context, 0 otherwise.")

class SyntheticQAUnit(BaseModel): question: str = Field(description="Synthesized question reflecting reasoning or specific facts.") ground_truth: str = Field(description="Factual ground-truth answer directly inferred from the context.")

class RAGEvaluationEngine: def init(self, openai_api_key: str): self.client = OpenAI(api_key=openai_api_key)

def generate_synthetic_dataset(self, document_chunks: List[str], num_samples: int = 5) -> List[Dict[str, Any]]:
    """
    Processes raw document chunks and leverages structured LLM parsing
    to compile a Golden Dataset of synthetic Questions & Ground Truth Answers.
    """
    dataset = []
    for i, chunk in enumerate(document_chunks[:num_samples]):
        prompt = (
            "You are an expert MLOps testing agent. Analyze the provided document chunk "
            "and generate a high-quality Question and a corresponding Ground Truth Answer "
            "based strictly on the facts inside the chunk. Do not assume or extrapolate outside the text."
        )

        response = self.client.beta.chat.completions.parse(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": prompt},
                {"role": "user", "content": f"Document Chunk:\n{chunk}"}
            ],
            response_format=SyntheticQAUnit,
            temperature=0.3
        )

        qa_unit = response.choices[0].message.parsed
        dataset.append({
            "id": f"syn_{i}",
            "question": qa_unit.question,
            "ground_truth": qa_unit.ground_truth,
            "gold_context": chunk
        })

    return dataset

def evaluate_faithfulness(self, context: str, answer: str) -> float:
    """
    Calculates Faithfulness score: F = (verified_statements) / (total_statements)
    """
    # Step A: Extract individual factual statements from answer
    extraction_prompt = "Extract all distinct factual statements made inside the provided answer text."
    extract_response = self.client.beta.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": extraction_prompt},
            {"role": "user", "content": f"Answer:\n{answer}"}
        ],
        response_format=StatementExtraction,
        temperature=0.0
    )
    statements = extract_response.choices[0].message.parsed.statements

    if not statements:
        return 1.0  # Safe default if no statements are asserted

    # Step B: Verify each statement against the retrieved context
    verified_count = 0
    for stmt in statements:
        verification_prompt = (
            "Verify whether the provided statement is fully supported by the retrieved context. "
            "Output 1 for verdict if the statement is supported, 0 otherwise."
        )
        verify_response = self.client.beta.chat.completions.parse(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": verification_prompt},
                {"role": "user", "content": f"Context:\n{context}\n\nStatement:\n{stmt}"}
            ],
            response_format=StatementVerification,
            temperature=0.0
        )

        verification = verify_response.choices[0].message.parsed
        if verification.verdict == 1:
            verified_count += 1

    # Step C: Compute Faithfulness Ratio
    return float(verified_count / len(statements))

def evaluate_answer_relevance(self, query: str, answer: str) -> float:
    """
    Calculates Answer Relevance via Cosine Similarity of query embeddings.
    """
    # Step A: Retrieve embeddings for query and answer
    def get_embedding(text: str) -> List[float]:
        response = self.client.embeddings.create(
            model="text-embedding-3-small",
            input=text
        )
        return response.data[0].embedding

    vec_query = np.array(get_embedding(query))
    vec_answer = np.array(get_embedding(answer))

    # Step B: Compute Cosine Similarity
    dot_product = np.dot(vec_query, vec_answer)
    norm_q = np.linalg.norm(vec_query)
    norm_a = np.linalg.norm(vec_answer)

    if norm_q == 0 or norm_a == 0:
        return 0.0

    return float(dot_product / (norm_q * norm_a))

1. Continuous Integration (CI) MLOps Pipeline Gate

To prevent regression during continuous deployment updates, we configure our automated CI/CD engine (e.g., GitHub Actions, GitLab CI) to execute this pytest suite. If any candidate model changes, prompt updates, or parser upgrades cause Faithfulness or Relevance to drop below our SLA baselines, the build pipeline exits with a non-zero code, blocking deployment.

import pytest

Baseline SLA configurations

THRESHOLD_FAITHFULNESS = 0.85 THRESHOLD_RELEVANCE = 0.80

@pytest.fixture def evaluator_engine(): # Fetch key securely from environment variable bounds api_key = os.getenv("OPENAI_API_KEY", "mock-key") return RAGEvaluationEngine(openai_api_key=api_key)

def test_retrieval_faithfulness_slas(evaluator_engine): # Retrieve a high-signal factual context context = "Project Aegis utilizes dual-channel fiber optical interconnects restricted to 10Gbps bounds."

# Candidate generator output containing a blatant hallucination (not supported in context)
hallucinated_answer = "Project Aegis uses fiber interconnects running at 100Gbps."

score = evaluator_engine.evaluate_faithfulness(context, hallucinated_answer)

# Assert that hallucinated answer falls below strict faithfulness threshold limits
assert score < THRESHOLD_FAITHFULNESS, f"Hallucinated answer scored too high: {score}"

def test_correct_answer_relevance(evaluator_engine): query = "What is the speed limit of Project Aegis interconnects?" correct_answer = "The interconnects are capped and restricted to a maximum threshold of 10Gbps."

score = evaluator_engine.evaluate_answer_relevance(query, correct_answer)

# Assert relevance meets acceptable baseline bounds
assert score >= THRESHOLD_RELEVANCE, f"Relevant answer scored too low: {score}"

1. Verification & Validation Protocols

6.1 Perplexity Preservation Test

Verify that the synthetic data generator only extracts assertions present in source documents, eliminating hallucinated Question-Answer targets from your golden reference sets.

def test_synthesizer_accuracy(): engine = RAGEvaluationEngine(openai_api_key=os.getenv("OPENAI_API_KEY", "mock-key")) document_chunks = [ "The server rack footprint requires exactly 4.2 square meters of workspace in the data center." ]

synthetic_dataset = engine.generate_synthetic_dataset(document_chunks, num_samples=1)

assert len(synthetic_dataset) == 1
assert "4.2" in synthetic_dataset[0]["ground_truth"]
assert "4.2" in synthetic_dataset[0]["question"] or "footprint" in synthetic_dataset[0]["question"].lower()

6.2 Offline Judge Execution Isolation

Assert that local execution models can run standard Judge evaluations (e.g., using a local vLLM endpoint overriding OPENAI_API_BASE), isolating testing workloads from public web metrics.

def test_evaluation_isolation(): # Setup custom base url targeting local mock loopback or vLLM container os.environ["OPENAI_API_BASE"] = "http://localhost:8000/v1"

engine = RAGEvaluationEngine(openai_api_key="local-token")
# Execute evaluating tasks, confirming no outbound exceptions occur
assert engine is not None

Required Technical Skills

Advanced Engineering Skills Inventory: Automated RAG Evaluation & Synthetic Testing (RAG-010)

This matrix details the advanced system engineering, mathematical formulation, MLOps orchestration, and telemetry-gathering capabilities required to design, construct, and scale continuous-integration evaluation pipelines for enterprise RAG platforms.

1. Multi-Tenant Evaluation Partitioning & Database Design

Evaluating concurrent LLM-as-a-judge workloads under multi-tenant structures requires isolating candidate runs, golden datasets, and execution traces programmatically.

                [ Multi-Tenant CI/CD Request ]
                              │
     ┌────────────────────────┴────────────────────────┐
     ▼                                                 ▼

[ PostgreSQL Storage ] [ Redis Prioritized Queue ] • Tenant Row-Level Security active • Round-Robin polling workers • Isolates Golden Datasets • Dynamic worker scaling

Relational Multi-Tenant Storage Architectures:

Implementing PostgreSQL database structures with Row-Level Security (RLS) constraints mapped to tenant JWT scopes to store evaluation records, golden datasets, and test parameters.

Creating index patterns over composite partitions (tenant_id, created_at) to optimize lookups of historic evaluations and track regression metrics across deployment logs.

Prioritized Job Routing Mechanics:

Configuring message broker loops using Redis and Celery to manage long-running multi-tenant evaluation tasks.

Writing custom round-robin scheduling queues that prioritize jobs dynamically, preventing large enterprise dataset evaluations from starving smaller tenants.

1. Unsupervised Synthetic QA Dataset Generation

Manually constructing golden test sets is slow and subjective. Scaling continuous validation demands automating the generation of diverse, high-signal, and statistically valid test suites.

Document Semantic Processing:

Parsing raw enterprise document collections (PDFs, Markdown, structural logs) and extracting target context windows using layout-aware chunking rules.

Applying semantic clustering algorithms to identify high-information chunks that represent ideal testing nodes.

Structured Multi-Type QA Synthesis:

Developing prompting sequences that instruct LLMs to output strict, structured JSON representing different question archetypes:

Simple Questions: Directly answered by single sentence assertions inside a retrieved context block.

Reasoning Questions: Requiring multi-step logical inference over the context space.

Multi-Context Questions: Compiling facts from separate, distinct document fragments to synthesize a ground-truth answer.

Formulating automated verification passes to ensure synthesized ground-truth answers are strictly grounded in the parent context chunk, discarding hallucinated test items programmatically.

1. Mathematical Formalization of RAG Triad Metrics

Evaluating RAG systems requires moving beyond qualitative assessment, modeling structural metrics using deterministic mathematical scoring metrics.

Let $Q$ represent the query, $C = {c_1, c_2, \dots, c_k}$ represent the retrieved context, $A$ represent the generated answer, and $G$ represent the gold-standard ground truth.

Faithfulness (Groundedness) Score ($F$):

Implementing multi-step semantic extraction loops. The evaluator LLM extracts a set of individual factual statements $S(A) = {s_1, s_2, \dots, s_N}$ from the candidate response. For each statement $s_i$, the system evaluates a binary verification index $V(s_i, C) \in {0, 1}$:

$$F = \frac{\sum_{i=1}^{N} V(s_i, C)}{N} \quad \text{where } N = \vert{}S(A)\vert{}$$

Answer Relevance ($AR$):

Generating a list of $M$ synthetic query permutations $Q' = {q'_1, q'_2, \dots, q'_M}$ based solely on the generated answer $A$. Computing cosine similarities of deep embeddings between original and synthetic queries to score target relevance:

$$AR = \frac{1}{M} \sum_{i=1}^{M} \frac{\mathbf{e}Q \cdot \mathbf{e}{q'i}^T}{\Vert{}\mathbf{e}_Q\Vert{} \Vert{}\mathbf{e}{q'_i}\Vert{}}$$

Context Recall ($CR$):

Splitting the ground-truth answer $G$ into individual sentence assertions $G_{\text{sentences}} = {g_1, g_2, \dots, g_P}$. Evaluating whether each $g_j$ is present or supported within retrieved context segments $C$:

$$CR = \frac{\sum_{j=1}^{P} V(g_j, C)}{P} \quad \text{where } P = \vert{}G_{\text{sentences}}\vert{}$$

Context Precision ($CP$):

Quantifying whether the most relevant context blocks are prioritized at the top of the retrieved chunk array using Precision at rank $k$ formulas:

$$ CP = \frac{\sum_{k=1}^{K} \left( P@k \cdot I(k) \right)}{\sum_{k=1}^{K} I(k)} $$

Where $I(k) \in {0, 1}$ is the relevance indicator of the $k$-th retrieved context chunk $c_k$ relative to the ground truth $G$, and $P@k = \frac{\sum_{i=1}^{k} I(i)}{k}$.

1. Local High-Throughput Judge Model Hosting (vLLM)

Using external APIs to run millions of evaluation tokens introduces severe latency and financial costs. Securing and optimizing evaluation requires deploying dedicated, private serving runtimes.

[ Celery Workers ] ──► [ Async gRPC Calls ] ──► [ local vLLM Node (FP8) ] • Continuous Batching • Guided JSON Schemas

Continuous Batching & Hardware Optimization:

Deploying small instruction-tuned models (such as Llama-3-8B-Instruct or Mistral-7B-Instruct at FP8 or AWQ quantization) on private cloud nodes (AWS EC2 Spot instances).

Optimizing vLLM execution parameters (gpu-memory-utilization, max-model-len, tensor-parallel-size) to maximize VRAM utilization without trigger OOM faults.

Guided Structured Outputs:

Utilizing vLLM's structured output properties (using guided json schemas or regex parsers) to force LLM judges to return deterministic JSON arrays, completely bypassing regex parsing failures during validation runs.

1. CI/CD Pipeline Gates & Tracing Telemetry

Integrating evaluation metrics directly into Git workflows guarantees that regressions are blocked automatically before code shifts to production.

[ Pull Request Commit ] ──► [ GitHub Action Runner ] ──► [ local Pytest Suite ] │ (Checks SLAs) ├─► Pass: Auto-Merge └─► Fail: Block Deploy

Automated CI/CD Verification Gates:

Constructing automated testing setups (via GitHub Actions, GitLab CI, or Jenkins) that execute pytest suites on code modification.

Writing non-zero exit gate configurations that programmatically terminate deploy processes if candidate pipelines register evaluation drops below baseline boundaries (e.g., $\Delta_{\text{metric}} < -0.05$).

Open-Telemetry Tracing Integration:

Instrumenting production retrieval loops using open-telemetry standards (utilizing libraries like Arize Phoenix or TruLens).

Capturing, serializing, and exporting nested execution traces (embedding latency, vector index recall time, LLM generation speed) to centralized monitoring dashboards for real-time trace debugging.

🚀 Quick Installation & Setup Guide

To run this project locally, follow these simple steps designed for both developers and users with no prior programming experience.

1. Download Python: Make sure you have Python 3.10+ installed on your computer. Download it from the official Python Website. During installation, make sure to check the box "Add Python to PATH".
2. Extract / Setup Code: Open the terminal (Command Prompt on Windows) and navigate to the project directory, or download/extract the project files.
3. Install Dependencies (pip): In Command Prompt, run the following command to install all necessary packages automatically:

   pip install -r requirements.txt

4. Setup API Keys / Environment: Open the .env file in a text editor (like Notepad) and paste your API keys:

   GEMINI_API_KEY=your_gemini_api_key_here

5. Launch the Application: Double-click the run_project.bat or run_pito.bat file, or run the following command:

   python app.py

   (or app_gui.py)