System Architecture Blueprint

RAG-007: Context Compression & Token Optimization — Engineering Blueprint

This specification details the mathematical formulations, system design, context distillation loops, and performance evaluation metrics for RAG-007 (Context Compression & Token Optimization Pipeline).

1. System Specifications & Performance SLA

Context Compression Latency: $< 40\text{ms}$ per 4,000 input tokens using a local quantized small language model (SLM) on a standard CPU/GPU worker node.

Token Reduction Ratio: Minimum $50\%$ to $60\%$ reduction of raw retrieved context tokens.

Semantic Preservation SLA: $> 95\%$ answers-fidelity retention (measured via semantic similarity comparison against raw, uncompressed generations).

Lost-in-the-Middle Mitigation: Direct placement optimization ensuring critical extracted nodes are concentrated in the top $10\%$ and bottom $10\%$ of the active prompt sequence.

1. Ingestion & Prompt-Compression Topology

In standard RAG, the retrieval step pulls raw documents and stuffs them directly into the prompt context. This floods the LLM with syntactic filler ("and", "the", repetitive boilerplate, redundant markdown tags). RAG-007 intercepts the retrieved chunk array and runs a compression sequence just-in-time before prompt assembly.

              [ Retrieved Document Chunks ]
                            │
                            ▼
          [ Dynamic Query-Aware Re-ordering ]
           • Push high-relevance chunks to extreme ends
           • Push lower-relevance chunks to center (U-shape)
                            │
                            ▼
          [ Local SLM Entropy Estimator (LLMLingua) ]
           • Compute perplexity score per token
           • Map information density distributions
                            │
                            ▼
          [ Information Bottleneck / Filter Gate ]
           • Drop tokens with low informational entropy
           • Filter out redundant boilerplate segments
                            │
     ┌──────────────────────┴──────────────────────┐
     ▼                                             ▼

[ Compressed Prompt Context ] [ Saved Token Metric ] • Bounded token budget • Up to 60% cost reduction • Sent to downstream LLM • Logged for SaaS billing

1. Mathematical Formalization of Prompt Compression

3.1 Token Entropy & Perplexity Budgeting

Let a retrieved document context $C$ be represented as a sequence of tokens $T = {t_1, t_2, \dots, t_N}$. We utilize a small helper language model $M_{\text{small}}$ to estimate the probability of each token $t_i$ given its prefix history $t_{<i}$:

$$P(t_i \mid t_{<i}, M_{\text{small}})$$

The information content (entropy) $I(t_i)$ of each individual token represents its level of surprise:

$$I(t_i) = -\log_2 P(t_i \mid t_{<i}, M_{\text{small}})$$

Predictable or highly repetitive syntactic structures (e.g., boilerplate notices, redundant conjunctions) have high probability scores, which translates to extremely low information entropy.

Given a compression token budget $B < N$, we formulate the compression sequence as an optimization problem where we seek to retain a subset of tokens $T' \subset T$ that maximizes cumulative information:

$$\arg\max_{T'} \sum_{t_k \in T'} I(t_k) \quad \text{subject to} \quad \vert{}T'\vert{} \le B$$

3.2 Bidirectional Re-ordering (Lost-in-the-Middle Correction)

LLMs exhibit U-shaped performance curves when retrieving data from long contexts: they recall information from the very beginning and very end of prompts with high accuracy, while forgetting facts buried in the middle.

Given retrieved document chunks $\mathcal{D} = {D_1, D_2, \dots, D_M}$ sorted descending by their initial retrieval similarity score, we map them into a bidirectional sequence $\mathcal{D}^*$ using a U-shaped index interleaving function:

$$\mathcal{D}^*k = \begin{cases} D_i, & \text{for } k = 2i \quad (\text{even indices placed at the beginning}) \ D{M - i}, & \text{for } k = 2i + 1 \quad (\text{odd indices placed at the end}) \end{cases}$$

This moves the highest-relevance documents to the outermost positions of the context window (maximizing retention) and pushes the low-relevance documents into the middle zone, where they undergo aggressive token compression.

1. Complete Implementation Specification

This self-contained Python module implements bidirectional re-ordering, calls a local Hugging Face transformer model to estimate token perplexity, filters out redundant syntax tokens, and yields compressed prompt contexts.

import os import torch from typing import List, Dict, Any, Tuple from transformers import AutoTokenizer, AutoModelForCausalLM from openai import OpenAI

class ContextCompressionPipeline: def init(self, small_model_name: str = "gpt2", openai_api_key: str = "mock-key"): # 1. Initialize local small model for token entropy estimation self.tokenizer = AutoTokenizer.from_pretrained(small_model_name) self.slm_model = AutoModelForCausalLM.from_pretrained(small_model_name)

    # Enforce evaluation mode
    self.slm_model.eval()
    if torch.cuda.is_available():
        self.slm_model = self.slm_model.to("cuda")

    # 2. Initialize downstream completion client
    self.openai_client = OpenAI(api_key=openai_api_key)

def u_shape_reorder(self, chunks: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
    """
    Reorders document chunks into a bidirectional U-shape to mitigate
    the 'Lost in the Middle' retrieval performance degradation.
    """
    # Assume input chunks are pre-sorted descending by similarity score
    sorted_chunks = sorted(chunks, key=lambda x: x.get("score", 0), reverse=True)
    reordered = [None] * len(sorted_chunks)

    left = 0
    right = len(sorted_chunks) - 1

    # Interleave elements: place highest at the start, next highest at the end, etc.
    for i in range(len(sorted_chunks)):
        if i % 2 == 0:
            reordered[left] = sorted_chunks[i]
            left += 1
        else:
            reordered[right] = sorted_chunks[i]
            right -= 1

    return reordered

def compute_token_entropies(self, text: str) -> List[Tuple[str, float]]:
    """
    Calculates the information entropy (surprise metric) of each token 
    using a local small language model.
    """
    inputs = self.tokenizer(text, return_tensors="pt")
    input_ids = inputs["input_ids"]

    if torch.cuda.is_available():
        input_ids = input_ids.to("cuda")

    with torch.no_grad():
        outputs = self.slm_model(input_ids)
        logits = outputs.logits # Shape: (1, seq_len, vocab_size)

    # Shift logits and labels to match history predictions
    shift_logits = logits[..., :-1, :].squeeze(0) # Shape: (seq_len - 1, vocab_size)
    shift_labels = input_ids[..., 1:].squeeze(0) # Shape: (seq_len - 1)

    # Compute softmax probabilities along vocabulary axis
    probs = torch.softmax(shift_logits, dim=-1)

    # Extract probabilities of the actual target tokens
    target_probs = probs[torch.arange(shift_labels.size(0)), shift_labels]

    # Information Content (Entropy) = -log2(probability)
    entropies = -torch.log2(target_probs).cpu().numpy()

    # Build list of token-entropy mappings (first token has no history, assigned average)
    tokens = [self.tokenizer.decode([tid]) for tid in input_ids[0]]
    token_entropy_pairs = [(tokens[0], float(np.mean(entropies)) if len(entropies) > 0 else 1.0)]

    for i, entropy in enumerate(entropies):
        token_entropy_pairs.append((tokens[i + 1], float(entropy)))

    return token_entropy_pairs

def compress_context(self, raw_context: str, keep_ratio: float = 0.50) -> Tuple[str, Dict[str, Any]]:
    """
    Strips low-entropy/predictable tokens to compress context size.
    """
    token_entropy_pairs = self.compute_token_entropies(raw_context)
    total_tokens = len(token_entropy_pairs)
    target_count = int(total_tokens * keep_ratio)

    # Sort tokens by entropy to establish threshold
    sorted_by_entropy = sorted(token_entropy_pairs, key=lambda x: x[1], reverse=True)
    threshold_score = sorted_by_entropy[target_count - 1][1] if target_count < total_tokens else 0.0

    # Build compressed token list retaining positional order
    compressed_tokens = []
    for token, entropy in token_entropy_pairs:
        # Always keep structure tokens/punctuation to preserve syntax reading
        if entropy >= threshold_score or token.strip() in [".", ",", "?", "!", "\n"]:
            compressed_tokens.append(token)

    compressed_text = "".join(compressed_tokens)

    metrics = {
        "original_token_count": total_tokens,
        "compressed_token_count": len(compressed_tokens),
        "compression_ratio": 1.0 - (len(compressed_tokens) / total_tokens),
        "threshold_entropy": threshold_score
    }

    return compressed_text, metrics

async def execute_optimized_query(self, query: str, retrieved_chunks: List[Dict[str, Any]]) -> str:
    """
    Executes end-to-end token-optimized query.
    """
    # Step 1: Reorder chunks to solve 'Lost in the Middle'
    reordered_chunks = self.u_shape_reorder(retrieved_chunks)
    raw_context = "\n\n".join([chunk["text"] for chunk in reordered_chunks])

    # Step 2: Apply dynamic context compression (keep 50%)
    compressed_context, metrics = self.compress_context(raw_context, keep_ratio=0.50)

    # Step 3: Run downstream generation on the condensed payload
    prompt = (
        f"Context Documents (Compressed):\n{compressed_context}\n\n"
        f"Query: {query}\n"
        f"Answer the query using only high-signal metrics from the context."
    )

    response = self.openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0
    )

    return response.choices[0].message.content

1. Verification & Validation Protocols

5.1 Compression Ratio & Integrity Test

We implement pytest validation to guarantee that the compression wrapper strips out predictable sequences while retaining critical named entities and numbers.

import pytest import numpy as np

def test_context_compression_retains_information(): pipeline = ContextCompressionPipeline(small_model_name="gpt2")

# Highly repetitive filler text with distinct critical numbers
raw_context = (
    "This is a boilerplate header. Indeed, we write many filler words in this text segment. "
    "The secret decryption key for the database is exactly 99281-X. "
    "As mentioned, boilerplate headers are very predictable and often repeated."
)

compressed, metrics = pipeline.compress_context(raw_context, keep_ratio=0.45)

# Assert compression target met
assert metrics["compression_ratio"] >= 0.50

# Assert critical, highly-surprising token was retained
assert "99281-X" in compressed

# Assert predictable boilerplate is heavily reduced
assert "boilerplate" not in compressed or compressed.count("boilerplate") < raw_context.count("boilerplate")

5.2 Bidirectional Ordering Verification

Verify that the interleaving routing works correctly to balance candidate documents.

def test_u_shape_reordering_positions(): pipeline = ContextCompressionPipeline(small_model_name="gpt2")

# Mock chunks sorted descending by relevance score (doc_0 is best, doc_4 is worst)
mock_chunks = [
    {"id": "doc_0", "score": 0.95, "text": "A"},
    {"id": "doc_1", "score": 0.85, "text": "B"},
    {"id": "doc_2", "score": 0.75, "text": "C"},
    {"id": "doc_3", "score": 0.65, "text": "D"},
    {"id": "doc_4", "score": 0.55, "text": "E"}
]

reordered = pipeline.u_shape_reorder(mock_chunks)

# Expected ordering of indices:
# Index 0 (doc_0) -> placed at reordered[0]
# Index 1 (doc_1) -> placed at reordered[4] (end)
# Index 2 (doc_2) -> placed at reordered[1]
# Index 3 (doc_3) -> placed at reordered[3]
# Index 4 (doc_4) -> placed at reordered[2] (center)

assert reordered[0]["id"] == "doc_0" # Top candidate at the front
assert reordered[-1]["id"] == "doc_1" # Second top candidate at the end
assert reordered[2]["id"] == "doc_4" # Worst candidate pushed into the middle zone

Required Technical Skills

Advanced Engineering Skills Inventory: Context Compression & Token Optimization (RAG-007)

This matrix details the specialized compiler engineering, information theory, deep learning serving, multi-tenant billing, and interactive visual interface skills required to design, deploy, and scale real-time token compression and context-optimization systems.

1. Information Theory & Token Perplexity Estimation

Unlike heuristic regex or keyword-based pruning, dynamic context distillation requires analyzing the underlying information distribution of prompt texts using probabilistic linguistic models.

              [ Raw Long-Context Documents ]
                            │
              [ Quantized local SLM Engine ]
             • Single Forward Pass (Zero Generation)
             • Computes Token Probability Distributions
                            │
                            ▼
           [ Token Surprisal (Information Density) ]
            • High Surprisal: Keep (Emerald)
            • Low Surprisal: Prune (Red / Strike)

Mathematical Entropy Foundations:

Estimating token-level information density (surprisal) using negative log probability distributions:

$$I(t_i) = -\log_2 P(t_i \mid t_{<i}, M_{\text{SLM}})$$

Calculating joint sequence perplexity thresholds to dynamically determine candidate token cutoff boundaries across varying target keep-ratios.

Formulating structure-preserving parsing exceptions (e.g., maintaining mathematical symbols, syntactic separators, and punctuation tokens regardless of information score) to prevent downstream context fragmentation.

Information Bottleneck Modeling:

Implementing lossy text compression strategies that filter out semantic filler words, boilerplate disclaimers, and grammatical redundant structures while retaining 100% of the core signal (numbers, named entities, key attributes).

1. Low-Latency local SLM Inference & vLLM Serving

Executing deep learning forward passes within a tight, interactive SLA ($< 40\text{ms}$) requires highly optimized execution runtime environments.

Continuous Batching & Serving Pipelines:

Configuring high-throughput model serving engines (such as vLLM or Triton Inference Server) to extract raw token logprobs.

Tuning batch metrics (tensor-parallel-size, gpu-memory-utilization) to run multi-tenant concurrent requests without triggering GPU Out-Of-Memory (OOM) exceptions.

Disabling autoregressive decoder generation loops (sampling, temperature controls) and executing strictly single forward passes across sequence frames to capture logits instantly.

Hardware Optimization & Quantization:

Quantizing large context models (such as LLMLingua-2 or Llama-3-8B) down to FP8 or AWQ formats to ensure fit in restricted GPU VRAM configurations while maintaining logprob accuracy.

Binding processing threads to dedicated CPU/GPU cores using process affinity tasksets to guarantee consistent execution times under intense traffic bursts.

1. "Lost-in-the-Middle" Retrieval Mitigation

Retrieving documents and appending them sequentially causes large language models to ignore facts situated in the middle of long prompts.

[ Input Documents sorted by relevance: D1, D2, D3, D4, D5 ] │ ▼ [ Bidirectional Interleaving Function (U-Shape) ] │ ▼ [ Final Prompt: D1 (Start) -> D3 -> D5 -> D4 -> D2 (End) ] • High-relevance at extreme boundaries (Maximum Retention) • Low-relevance pushed to center (Targets of Aggressive Pruning)

Bidirectional Interleaving Formulations:

Implementing U-shaped indexing functions to reconstruct candidate context lists, moving high-relevance elements to the absolute boundaries (first 10% and last 10% of the prompt window) where attention weights are mathematically highest:

$$\mathcal{D}^*k = \begin{cases} D_i, & \text{for } k = 2i \ D{M - i}, & \text{for } k = 2i + 1 \end{cases}$$

Dynamically scaling pruning rates based on document position—compressing documents pushed to the middle of the sequence far more aggressively than documents anchored at the extreme ends.

1. Multi-Tenant Metrology & Cost Auditing

When deploying token-reduction systems in commercial B2B environments, tracking, auditing, and billing customer token utilization is crucial to maintaining software gross margins.

Relational Metrology Logging:

Designing transaction-safe database architectures (PostgreSQL) to record, aggregate, and index raw-vs-compressed token footprints atomically on every API execution.

Creating high-speed database indexes over composite partition boundaries (tenant_id, created_at) to support rapid usage-based billing computations.

Dynamic SaaS Quota Enforcement:

Building sliding-window quota filters in Redis to enforce customized daily/monthly tenant token maximums.

Designing graceful degradation fallbacks to bypass the SLM engine and return raw uncompressed contexts to the downstream LLM if system queues experience high congestion.

1. Next.js Dynamic UI Token Rendering

Visualizing complex, token-level calculations requires building performant frontend layouts that can render color-coded token weights without introducing browser rendering lag.

High-Speed Memoization & Dynamic Styling:

Implementing highly optimized React state hook configurations (useMemo) to compute dynamic threshold cuts across thousands of active tokens in real-time.

Utilizing Tailwind CSS to dynamically transition CSS classes (strikethroughs, text opacity shifts, background shading) on token coordinates based on real-time slider updates.

Building responsive SVG or custom canvas graphs to render monthly savings metrics, system execution latency logs, and relative compression rates.

🚀 Quick Installation & Setup Guide

To run this project locally, follow these simple steps designed for both developers and users with no prior programming experience.

1. Download Python: Make sure you have Python 3.10+ installed on your computer. Download it from the official Python Website. During installation, make sure to check the box "Add Python to PATH".
2. Extract / Setup Code: Open the terminal (Command Prompt on Windows) and navigate to the project directory, or download/extract the project files.
3. Install Dependencies (pip): In Command Prompt, run the following command to install all necessary packages automatically:

   pip install -r requirements.txt

4. Setup API Keys / Environment: Open the .env file in a text editor (like Notepad) and paste your API keys:

   GEMINI_API_KEY=your_gemini_api_key_here

5. Launch the Application: Double-click the run_project.bat or run_pito.bat file, or run the following command:

   python app.py

   (or app_gui.py)