System Architecture Blueprint

RAG-002: Multimodal Enterprise Document RAG — Engineering Blueprint

This specification details the end-to-end architecture, mathematical concepts, indexing strategies, and retrieval configurations for RAG-002 (Multimodal Enterprise Document RAG).

1. System Specifications & Performance SLA

Target Ingest Throughput: $< 2.0\text{ seconds}$ per page (including PDF-to-image conversion, LlamaParse layout extraction, and ColPali multi-vector generation).

Retrieval SLA (Prefetch + MaxSim Rerank): $< 80\text{ms}$ over $100,000$ indexed pages.

Accuracy Target: $> 95\%$ layout and table retrieval precision, resolving complex structural alignments (e.g., cell coordinate mapping, nested charts, dual-column blueprints).

1. Ingestion & Storage Architecture

Traditional parsing approaches strip visual context (fonts, column boundaries, embedded illustrations), destroying semantic relationships. RAG-002 implements a Dual-Track Ingestion Engine:

                   [ Incoming Enterprise PDF Document ]
                                    │
                ┌───────────────────┴───────────────────┐
                ▼                                       ▼
    [ Path A: Visual Track ]               [ Path B: Structural Track ]
     Render PDF Page to Image               Analyze Layout via LlamaParse
                │                                       │
                ▼                                       ▼
     [ ColPali/ColQwen2.5 ]                  • Markdown Layouts
   Extract 1024 Patch Vectors                • Structural Table Markdowns
                │                            • High-Res Image Crops (Charts)
                ▼                                       │
     [ Training-free Pooling ]                          │
       Reduce to 32 Vectors                             │
                │                                       │
                ▼                                       ▼
   [ Qdrant Vector Collection ] <───────────────────────┘
    • Named Vectors:                               Payload Links:
      - "pooled_vector" (HNSW)                      - raw_image_b64
      - "full_vector" (exact)                       - markdown_tables

2.1 Qdrant Multi-Vector named Collection Setup

Storing $1024$ embeddings (dimension $128$) per document page scales poorly in memory and compute. To mitigate this, we configure a Multi-Stage Named Vector Collection in Qdrant:

pooled_vector: A compressed representation (32 vectors of dimension 128) used for fast, first-stage HNSW candidate generation.

full_vector: The original un-pooled representation (1024 vectors of dimension 128) used for second-stage exact late-interaction reranking.

{ "name": "multimodal_enterprise_kb", "vectors": { "pooled_vector": { "size": 128, "distance": "Cosine", "multivector_config": { "comparator": "MaxSim" }, "hnsw_config": { "m": 16, "ef_construct": 128 } }, "full_vector": { "size": 128, "distance": "Cosine", "multivector_config": { "comparator": "MaxSim" }, "quantization_config": { "binary": { "always_ram": true } } } } }

1. Mathematical Retrieval Formulation

3.1 ColPali Late Interaction Model

ColPali projects Vision Transformer (ViT) patch representations (output from a contrastive Vision-Language Model like PaliGemma or Qwen2-VL) into a low-dimensional vector space.

Given a text query $Q$ tokenized into sequence length $\vert{}Q\vert{}$, and a document page image $D$ split into patches of size $32 \times 32 = 1024$ (yields sequence length $\vert{}D\vert{}$), the similarity score is computed using the Late Interaction MaxSim operator:

$$Score(Q, D) = \sum_{i=1}^{\vert{}Q\vert{}} \max_{j=1}^{\vert{}D\vert{}} \left( q_i \cdot d_j^T \right)$$

Where:

$q_i \in \mathbb{R}^{128}$ is the normalized embedding vector of the $i$-th query token.

$d_j \in \mathbb{R}^{128}$ is the normalized embedding vector of the $j$-th document patch.

$(\cdot)$ represents the dot product.

3.2 Training-Free Spatial Pooling

To drastically speed up candidate generation (first-stage retrieval), the $1024$ patch embeddings (which structurally correspond to a $32 \times 32$ spatial tile matrix) are pooled row-wise or in sub-grids:

$$\mathbf{P}r = \text{Mean}\left(\mathbf{d}{(r-1)\cdot{}k + 1}, \mathbf{d}{(r-1)\cdot{}k + 2}, \dots, \mathbf{d}{r\cdot{}k}\right)$$

Where:

$k$ is the pooling stride factor (e.g., $32$).

$\mathbf{P}_r \in \mathbb{R}^{128}$ represents the compressed $r$-th representative vector (scaling the representation from $1024$ vectors down to $32$).

1. Multimodal Orchestrator Implementation

This Python module implements the parallel ingestion pipelines, spatial pooling utilities, two-stage Qdrant retrieval queries, and vision-model synthesis.

import os import torch import asyncio import numpy as np from PIL import Image from typing import List, Dict, Any, Tuple from colpali_engine.models import ColPali, ColPaliProcessor from qdrant_client import AsyncQdrantClient from qdrant_client.models import PointStruct, VectorInput from openai import AsyncOpenAI # For Vision LLM (GPT-4o or Qwen2-VL API)

class MultimodalRAGEngine: def init(self, qdrant_url: str, qdrant_api_key: str, hf_token: str): # 1. Initialize Clients self.qdrant_client = AsyncQdrantClient(url=qdrant_url, api_key=qdrant_api_key) self.llm_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))

    # 2. Initialize ColPali Model on Local Accelerator (CUDA / MPS)
    self.device = "cuda" if torch.cuda.is_available() else "cpu"
    self.dtype = torch.bfloat16 if self.device == "cuda" else torch.float32

    self.model = ColPali.from_pretrained(
        "vidore/colpali-v1.3",
        torch_dtype=self.dtype,
        device_map=self.device,
        token=hf_token
    ).eval()

    self.processor = ColPaliProcessor.from_pretrained("vidore/colpali-v1.3")

def _spatial_mean_pooling(self, embeddings: torch.Tensor, target_vectors: int = 32) -> np.ndarray:
    """
    Compresses 1024 visual tokens down to 32 named vectors
    using a training-free 1D sequence mean-pooling strategy.
    """
    emb_np = embeddings.detach().cpu().to(torch.float32).numpy() # Shape: (1024, 128)
    # Reshape to (32, 32, 128) to group spatially by row segments
    reshaped = emb_np.reshape(target_vectors, -1, 128)
    # Average along the segment axis -> output shape: (32, 128)
    pooled = np.mean(reshaped, axis=1)
    # Normalize vectors
    norms = np.linalg.norm(pooled, axis=1, keepdims=True)
    return (pooled / np.where(norms == 0, 1e-12, norms)).astype(np.float32)

async def ingest_page(self, page_image: Image.Image, text_layout_markdown: str, page_number: int, doc_id: str):
    """
    Dual Ingestion Step. Generates visual embeddings via ColPali and indexes
    structural markdown payloads returned from parsing.
    """
    # Step 1: Process visual image through ColPali
    processed_img = self.processor.process_images([page_image])
    processed_img = {k: v.to(self.model.device) for k, v in processed_img.items()}

    with torch.no_grad():
        # Get multi-vector representation: shape (1, 1024, 128)
        raw_embeddings = self.model(**processed_img)
        page_emb = raw_embeddings[0]

    # Step 2: Apply Spatial Mean Pooling to get fast-path vector
    pooled_emb = self._spatial_mean_pooling(page_emb, target_vectors=32)
    full_emb = page_emb.detach().cpu().to(torch.float32).numpy()

    # Step 3: Package payload and insert into Qdrant
    point_id = f"{doc_id}_{page_number}"
    point = PointStruct(
        id=point_id,
        vector={
            "pooled_vector": pooled_emb.tolist(),
            "full_vector": full_emb.tolist()
        },
        payload={
            "doc_id": doc_id,
            "page": page_number,
            "layout_markdown": text_layout_markdown,
            # Store structural markdown tables specifically
            "extracted_tables": [text_layout_markdown] # Filtered tables placeholder
        }
    )

    await self.qdrant_client.upsert(
        collection_name="multimodal_enterprise_kb",
        points=[point]
    )

async def retrieve(self, query: str, top_k: int = 3) -> List[Dict[str, Any]]:
    """
    Two-stage retrieval pipeline:
    1. Prefetch Top-30 results using fast HNSW on pooled vectors.
    2. Rerank down to top_k using exact MaxSim scoring on full vectors.
    """
    # Step 1: Embed Query via ColPali
    processed_query = self.processor.process_queries([query])
    processed_query = {k: v.to(self.model.device) for k, v in processed_query.items()}

    with torch.no_grad():
        query_emb = self.model(**processed_query)[0].detach().cpu().to(torch.float32).numpy()

    # Step 2: Stage 1 Search (Fast HNSW using compressed 32x128 representation)
    pref_results = await self.qdrant_client.query_points(
        collection_name="multimodal_enterprise_kb",
        query=query_emb.tolist(),
        using="pooled_vector",
        limit=30,
        with_payload=True,
        with_vectors=True
    )

    # Step 3: Stage 2 Exact Reranking using Full 1024x128 representation
    scored_candidates = []
    for point in pref_results.points:
        full_vectors = np.array(point.vectors["full_vector"]) # Shape (1024, 128)

        # Compute MaxSim Score manually: Sum_i ( Max_j ( q_i . d_j ) )
        # Query embedding shape: (len_q, 128)
        # Doc embedding shape: (1024, 128)
        dot_product = np.dot(query_emb, full_vectors.T) # Shape: (len_q, 1024)
        max_similarities = np.max(dot_product, axis=1) # Shape: (len_q,)
        max_sim_score = float(np.sum(max_similarities))

        scored_candidates.append({
            "id": point.id,
            "score": max_sim_score,
            "payload": point.payload
        })

    # Sort by exact late interaction score descending
    scored_candidates.sort(key=lambda x: x["score"], reverse=True)
    return scored_candidates[:top_k]

async def generate_response(self, query: str, retrieved_pages: List[Dict[str, Any]], page_images: List[Image.Image]) -> str:
    """
    Synthesizes the final output by presenting the raw, un-OCR'd page images 
    and high-fidelity tables to a vision LLM.
    """
    # Synthesize prompt with context markdowns
    markdown_contexts = "\n\n".join([
        f"--- PAGE {item['payload']['page']} --- \n{item['payload']['layout_markdown']}" 
        for item in retrieved_pages
    ])

    system_instructions = (
        "You are an expert enterprise operations analyst. Synthesize a precise, factual answer "
        "using the structural Markdown text AND the raw screenshot page attachments. "
        "If a table coordinate or value is referenced, verify it visually on the image page "
        "to prevent transcription and OCR hallucinations. Do not write generic assumptions."
    )

    # Assemble OpenAI Vision format message payload
    messages = [
        {"role": "system", "content": system_instructions},
        {"role": "user", "content": [
            {"type": "text", "text": f"Query: {query}\n\nRetrieved Structured Tables:\n{markdown_contexts}"}
        ]}
    ]

    # Append retrieved page images to prompt payload
    for i, img in enumerate(page_images[:len(retrieved_pages)]):
        # Base64-encode PIL image (helper logic omitted for brevity)
        import io, base64
        buffered = io.BytesIO()
        img.save(buffered, format="JPEG")
        img_b64 = base64.b64encode(buffered.getvalue()).decode('utf-8')

        messages[1]["content"].append({
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}
        })

    response = await self.llm_client.chat.completions.create(
        model="gpt-4o",  # Or local Qwen2-VL endpoints
        messages=messages,
        temperature=0.0,
        max_tokens=800
    )
    return response.choices[0].message.content

1. Verification & Validation Protocols

5.1 Layout Preservation Testing

To assert layout fidelity, run validation routines against test PDF matrices containing rotated structures and diverse formatting templates.

def test_table_coordinate_matching(): # Setup test page index with explicit coordinates extracted_md = "| Quarter | Revenue (B) |\n|---|---|\n| Q1 | $14.2B |\n| Q2 | $15.8B |" # Assert query matches coordinates structurally assert "Quarter" in extracted_md assert "$14.2B" in extracted_md

5.2 Offline Network Boundaries Validation

Confirm that all model operations execute local CUDA tensor logic rather than shipping image payloads outside your secure environment.

def test_retriever_offline_isolation(engine: MultimodalRAGEngine): # Disable outgoing internet access during execution import socket def block_sockets(args, *kwargs): raise socket.error("Network interface disconnected.")

socket.socket = block_sockets

# Run a test query. ColPali inference must complete entirely on-device
test_query = "What is the total quarterly operating income?"
results = asyncio.run(engine.retrieve(test_query, top_k=1))
assert len(results) > 0

Required Technical Skills

Advanced Engineering Skills Inventory: Multimodal Enterprise Document RAG (RAG-002)

This matrix maps out the advanced systems engineering, computer vision, deep learning, distributed processing, and frontend visualization competencies required to design, implement, and maintain high-fidelity multimodal retrieval pipelines.

1. Vision-Language Models & Late Interaction Architectures

To build retrieval engines that bypass the limitations of optical character recognition (OCR) and text-only embeddings, you must understand how visual representations are projected and queried in joint spaces.

              [ Raw PDF Page / Blueprints ]
                            │
             [ Vision Transformer (ViT) ]
                            │
   [ Low-Dimensional Token Projection (ColPali/ColQwen) ]
                            │
       [ Late Interaction MaxSim Math Evaluation ]

ColPali & ColQwen Model Architecture:

Fine-tuning and deploying Vision-Language Models (VLMs) like PaliGemma or Qwen2-VL for semantic document retrieval.

Understanding how Vision Transformers (ViTs) generate patch embeddings (e.g., mapping a page image to a grid of $32 \times 32 = 1024$ spatial patches).

Configuring custom projection layers that map high-dimensional transformer outputs into compact low-dimensional dense vectors (typically $d = 128$).

Late Interaction Operator (MaxSim):

Implementing and optimizing the $MaxSim$ alignment metric between a tokenized text query $Q$ and a patch-tokenized document image $D$:

$$Score(Q, D) = \sum_{i=1}^{\vert{}Q\vert{}} \max_{j=1}^{\vert{}D\vert{}} \left( q_i \cdot d_j^T \right)$$

Leveraging PyTorch tensor broadcasting to compute pairwise matrix multiplications efficiently across batch dimensions.

1. Advanced Multi-Vector Database Engineering

Storing multiple vectors per document page introduces severe memory and search complexity. Optimizing this storage footprint requires specialized indexing and compression techniques.

Multi-Vector Search Spaces (Qdrant / Milvus):

Configuring named vector collections designed to store arrays of dynamic-sized vector pools ($1024 \times 128$ dimensions per page) under single payload documents.

Managing search operations targeting multi-vector comparators (e.g., using Qdrant’s native MaxSim configuration settings).

Dimensionality Reduction & Pooling:

Formulating training-free spatial pooling strategies (such as row-wise mean pooling or localized $2\text{D}$ sub-grid max pooling) to compress page matrices from $1024$ down to $32$ representative vectors.

$$\mathbf{P}r = \text{Mean}\left(\mathbf{d}{(r-1)\cdot{}k + 1}, \mathbf{d}{(r-1)\cdot{}k + 2}, \dots, \mathbf{d}{r\cdot{}k}\right)$$

Executing two-stage search configurations: performing fast candidate pre-filtering (using HNSW on pooled vectors) before running exact $MaxSim$ scoring on the full patch matrix.

Quantization Protocols:

Setting up Binary Quantization (BQ) or Scalar Quantization (SQ) over patch matrices to run vector searches inside RAM without sacrificing the retrieval accuracy of the floating-point models.

1. High-Fidelity Layout Extraction & Computer Vision

Extracting coordinate structures, nested charts, and complex tables requires orchestrating traditional computer vision libraries alongside visual semantic document parsers.

Structural Parsing (LlamaParse):

Configuring multi-stage parse pipelines to output structural Markdown tables alongside layout coordinate bounding boxes.

Mapping parsed metadata attributes back to absolute pixel offsets for target image slices.

Image Preprocessing & Manipulation (OpenCV / PIL):

Writing pre-processing routines for document normalization, including rotation/skew correction, contrast adjustment (CLAHE), and image resizing with strict aspect-ratio conservation.

Executing crop-slice coordinates over targeted segments (extracting nested charts, diagrams, or tables) to minimize vision-LLM processing token overheads.

Converting PDF files to high-resolution rasterized images (using libraries like pdf2image or fitz/PyMuPDF) at optimal DPI metrics ($150\text{ to }300\text{ DPI}$) for visual encoders.

1. High-Concurrency Asynchronous Infrastructure

Because multimodal operations (PDF-to-image rasterization, VLM tensor generation, LlamaParse extraction) are highly resource-intensive ($1.5s \text{ to } 4s$ per page), synchronous API thread execution is a critical bottleneck.

[ FastAPI HTTP Endpoint ] ──► [ Decoupled Task Queue (Celery/RabbitMQ) ] │ ┌────────────────┴────────────────┐ ▼ ▼ [ GPU Workers ] [ CPU Workers ] ColPali Tensor Inferences LlamaParse Document Runs

Decoupled Worker Pipelines:

Architecting distributed, asynchronous task running architectures using Celery, Redis, or Temporal to handle long-running document ingestion.

Managing task-state updates in relational database structures (PostgreSQL) and pushing progress steps to frontends using Server-Sent Events (SSE) or WebSockets.

GPU Compute Optimization (CUDA / Triton):

Deploying and scaling models using PyTorch on CUDA environments with mixed-precision arithmetic configurations (float16, bfloat16, or int8 quantization).

Utilizing deep learning runtime engines like Triton Inference Server or vLLM to serve heavy multimodal visual models with high throughput and continuous batching capabilities.

1. Full-Stack Coordinate Rendering & Interactive Canvas

Displaying retrieved multimodal chunks requires mapping abstract mathematical percentages from backend vectors onto dynamic, responsive frontend canvases.

Dynamic HTML5 Canvas Orchestration (Next.js / React):

Building customized React hook configurations designed to consume dynamic coordinate data overlays.

Using the HTML5 Canvas $2\text{D}$ Context API to paint background page images and dynamic highlights concurrently.

Scaling abstract coordinate formats (such as percentage arrays $[0.0, 100.0]$ or normalized spans $[0.0, 1.0]$) to match dynamic browser container viewport ratios without losing alignment fidelity:

$$X_{\text{canvas}} = \left(\frac{X_{\text{box}}}{100}\right) \times W_{\text{canvas}}$$

Event-Driven UI Overlays:

Implementing hover intersections, bounding-box tooltips, and interactive sidebar lists linked to specific highlighted coordinates on active document layouts.

🚀 Quick Installation & Setup Guide

To run this project locally, follow these simple steps designed for both developers and users with no prior programming experience.

1. Download Python: Make sure you have Python 3.10+ installed on your computer. Download it from the official Python Website. During installation, make sure to check the box "Add Python to PATH".
2. Extract / Setup Code: Open the terminal (Command Prompt on Windows) and navigate to the project directory, or download/extract the project files.
3. Install Dependencies (pip): In Command Prompt, run the following command to install all necessary packages automatically:

   pip install -r requirements.txt

4. Setup API Keys / Environment: Open the .env file in a text editor (like Notepad) and paste your API keys:

   GEMINI_API_KEY=your_gemini_api_key_here

5. Launch the Application: Double-click the run_project.bat or run_pito.bat file, or run the following command:

   python app.py

   (or app_gui.py)