{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-zswAqxPwYaR"
   },
   "source": [
    "# ColPali retrieval: one run for the website demo\n",
    "\n",
    "This notebook runs **my ColPali retrieval application's own code** once, on a fixed set of public documents, and saves the results for the demo on my website. Visitors replay these results; they never run a model.\n",
    "\n",
    "**From the application, unchanged:** PDF pages to images (`PDFProcessor`), indexing every page as an image with ColQwen2 (`EmbeddingProcessor.embed_documents`), late-interaction scoring (`sum_max_sim`), the query pipeline with Gemini answers and similarity heat maps (`QueryProcessor.process_query`, `QueryVisualizer`).\n",
    "\n",
    "**Added for the demo only:** choosing the 33 pages and 14 questions, and saving the results (`run_demo.py`, below).\n",
    "\n",
    "**Adjusted at run time (the application's files are not edited):** the ColQwen2 checkpoint is pinned to its March 2025 snapshot (the model repository moved to a new format in August 2026); the Gemini model is `gemini-2.5-flash` (`gemini-2.0-flash-exp` was retired); the model runs on the GPU.\n",
    "\n",
    "**Documents:** US federal publications, in the public domain: 19 pages of the *NIST AI Risk Management Framework 1.0* and 14 slides of the *EIA Annual Energy Outlook 2026* presentation.\n",
    "\n",
    "### How to run it\n",
    "1. Runtime \u2192 Change runtime type \u2192 **L4**, **A100**, **H100** or **T4** GPU (not a TPU, not a Blackwell/G4 GPU).\n",
    "2. Key icon on the left \u2192 add a secret named `GEMINI_API_KEY` with a Gemini API key, and allow notebook access. (Without it, everything except the answers still runs.)\n",
    "3. Runtime \u2192 **Run all**. The first cell asks you to upload `colpali-retrieval-src.zip` (the application's `src` folder).\n",
    "4. About 15 minutes later the last cell downloads `colpali_export.zip`."
   ],
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VXDpRQQdwYaW"
   },
   "source": [
    "## 1. The application code"
   ],
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "FNgcWjVcwYaX"
   },
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Saving colpali-retrieval-src.zip to colpali-retrieval-src.zip\n",
      "application modules: ['__init__.py', 'api', 'cli', 'config', 'embeddings', 'models', 'querying', 'static', 'utils']\n"
     ]
    }
   ],
   "source": [
    "import os, zipfile\n",
    "os.makedirs(\"/content/app\", exist_ok=True)\n",
    "if not os.path.exists(\"/content/app/src\"):\n",
    "    from google.colab import files\n",
    "    uploaded = files.upload()  # choose colpali-retrieval-src.zip\n",
    "    with zipfile.ZipFile(next(iter(uploaded))) as z:\n",
    "        z.extractall(\"/content/app\")\n",
    "print(\"application modules:\", sorted(os.listdir(\"/content/app/src\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cd8htZLbwYaZ"
   },
   "source": [
    "## 2. A separate Python 3.11 environment with the application's 2025 package versions\n",
    "\n",
    "Colab's own environment (Python 3.13 and the newest TensorFlow, protobuf and transformers) clashes with the versions the application was built on. So the run happens in its own environment, created here with `uv`, and Colab's packages are left alone. This takes about 3 minutes."
   ],
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "o0ohPFkxwYaZ"
   },
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Writing /content/env-requirements.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile /content/env-requirements.txt\n",
    "# The ColPali retrieval app's dependencies, as in its requirements.txt (early 2025),\n",
    "# plus the versions they resolved to then. Installed into a separate Python 3.11\n",
    "# environment so nothing clashes with Colab's preinstalled packages.\n",
    "colpali-engine==0.3.6\n",
    "transformers==4.46.3\n",
    "tokenizers==0.20.3\n",
    "peft==0.13.2\n",
    "accelerate==1.0.1\n",
    "huggingface-hub==0.26.5\n",
    "safetensors==0.4.5\n",
    "numpy==1.26.4\n",
    "pillow==11.1.0\n",
    "google-generativeai==0.8.3\n",
    "pdf2image==1.17.0\n",
    "pypdf==5.1.0\n",
    "python-dotenv==1.0.1\n",
    "tqdm==4.67.1\n",
    "matplotlib==3.10.0\n",
    "seaborn==0.13.2\n",
    "fastapi==0.104.1\n",
    "python-multipart==0.0.6\n",
    "aiohttp==3.9.3\n",
    "sqlalchemy==2.0.36"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "LVOv2Vw9wYaZ"
   },
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "pdftoppm version 24.02.0\n",
      "Python 3.11.18 | torch 2.5.1+cu121 | transformers 4.46.3 | GPU: NVIDIA L4\n"
     ]
    }
   ],
   "source": [
    "!pip install -q uv\n",
    "!uv venv -q --allow-existing /content/env --python 3.11\n",
    "!uv pip install -q --python /content/env/bin/python torch==2.5.1 --index-url https://download.pytorch.org/whl/cu121\n",
    "!uv pip install -q --python /content/env/bin/python -r /content/env-requirements.txt\n",
    "!apt-get update -qq > /dev/null 2>&1; apt-get install -y -qq poppler-utils > /dev/null 2>&1; pdftoppm -v 2>&1 | head -1\n",
    "!/content/env/bin/python -c \"import sys, torch, transformers, colpali_engine; print('Python', sys.version.split()[0], '| torch', torch.__version__, '| transformers', transformers.__version__, '| GPU:', torch.cuda.get_device_name() if torch.cuda.is_available() else 'none')\"\n",
    "import subprocess\n",
    "cap = subprocess.run([\"/content/env/bin/python\", \"-c\", \"import torch; print(torch.cuda.get_device_capability()[0] if torch.cuda.is_available() else -1)\"], capture_output=True, text=True).stdout.strip()\n",
    "if cap == \"-1\":\n",
    "    print(\"No GPU: it will still run, but on the CPU it takes about an hour. For speed: Runtime \u2192 Change runtime type \u2192 L4/A100/T4 GPU.\")\n",
    "elif cap and int(cap) >= 10:\n",
    "    raise SystemExit(\"This GPU is too new for torch 2.5.1. Runtime \u2192 Change runtime type \u2192 L4, A100, H100 or T4.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1lQM1rvZwYaa"
   },
   "source": [
    "## 3. The run: the application's pipeline on the chosen pages and questions"
   ],
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "aZdEzzDSwYaa"
   },
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Writing /content/app/run_demo.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile /content/app/run_demo.py\n",
    "\"\"\"One run of the ColPali retrieval application's own code, for the website demo.\n",
    "\n",
    "Runs from the application's folder (the one that contains `src/`). Everything that\n",
    "matters comes from the application: PDFProcessor (PDF pages to images),\n",
    "EmbeddingProcessor (indexing with ColQwen2), sum_max_sim (late-interaction\n",
    "scoring), QueryProcessor.process_query (top pages, Gemini answers, heat maps) and\n",
    "QueryVisualizer. This script only chooses the documents and questions and saves\n",
    "the results.\n",
    "\n",
    "Three settings are adjusted at run time; the application's files are not edited:\n",
    "  1. the ColQwen2 checkpoint is pinned to its March 2025 snapshot (the model\n",
    "     repository moved to a new format in August 2026);\n",
    "  2. the Gemini model is gemini-2.5-flash (gemini-2.0-flash-exp was retired);\n",
    "  3. the model runs on the GPU when there is one.\n",
    "A Gemini call that fails is recorded as \"no answer\" so the run always finishes.\n",
    "\n",
    "Environment: GEMINI_API_KEY (optional), OUT (default ./colpali_export),\n",
    "MODEL_PATH (default: the pinned snapshot), MAX_PAGES / MAX_QUESTIONS (testing).\n",
    "\"\"\"\n",
    "\n",
    "import io\n",
    "import json\n",
    "import os\n",
    "import shutil\n",
    "import sys\n",
    "import time\n",
    "import urllib.request\n",
    "import zipfile\n",
    "\n",
    "sys.path.insert(0, os.getcwd())\n",
    "\n",
    "\n",
    "class Redact:\n",
    "    \"\"\"Hides the API key in everything printed or logged (some error messages include it).\"\"\"\n",
    "\n",
    "    def __init__(self, stream, secret):\n",
    "        self.stream, self.secret = stream, secret\n",
    "\n",
    "    def write(self, s):\n",
    "        return self.stream.write(s.replace(self.secret, \"***\") if self.secret else s)\n",
    "\n",
    "    def __getattr__(self, name):\n",
    "        return getattr(self.stream, name)\n",
    "\n",
    "\n",
    "# Before anything from the application is imported: its loggers keep a reference to\n",
    "# the output stream they are created with, and some error messages contain the key.\n",
    "if os.environ.get(\"GEMINI_API_KEY\"):\n",
    "    sys.stdout = Redact(sys.stdout, os.environ[\"GEMINI_API_KEY\"])\n",
    "    sys.stderr = Redact(sys.stderr, os.environ[\"GEMINI_API_KEY\"])\n",
    "\n",
    "import torch  # noqa: E402\n",
    "from pypdf import PdfReader, PdfWriter  # noqa: E402\n",
    "\n",
    "import src.config.settings as settings  # noqa: E402\n",
    "import src.models.initializers as initializers  # noqa: E402\n",
    "from src.embeddings.embedding_processor import EmbeddingProcessor  # noqa: E402\n",
    "from src.querying.query_processor import QueryProcessor  # noqa: E402\n",
    "from src.utils.pdf_utils import PDFProcessor  # noqa: E402\n",
    "\n",
    "USER_ID = 1  # the application keeps each user's documents apart; the demo is one user\n",
    "DOCS = [\n",
    "    {\"id\": \"nist\", \"file\": \"nist-ai-rmf-selected-pages.pdf\",\n",
    "     \"title\": \"NIST AI 100-1: Artificial Intelligence Risk Management Framework (AI RMF 1.0)\",\n",
    "     \"publisher\": \"National Institute of Standards and Technology, January 2023\",\n",
    "     \"url\": \"https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf\",\n",
    "     \"pages\": [6, 7, 8, 9, 10, 15, 16, 17, 19, 20, 21, 22, 23, 25, 27, 31, 34, 37, 43]},\n",
    "    {\"id\": \"eia\", \"file\": \"eia-aeo2026-slides-2-15.pdf\",\n",
    "     \"title\": \"Annual Energy Outlook 2026 (release presentation)\",\n",
    "     \"publisher\": \"U.S. Energy Information Administration, April 2026\",\n",
    "     \"url\": \"https://www.eia.gov/outlooks/aeo/pdf/AEO2026_Release_Presentation.pdf\",\n",
    "     \"pages\": list(range(2, 16))},\n",
    "]\n",
    "QUESTIONS = [\n",
    "    \"Which end-use sector's energy consumption grows the most by 2050?\",\n",
    "    \"How much electricity could data center servers use in 2050 in the High Electricity Demand case?\",\n",
    "    \"What share of light-duty vehicles could be battery electric by 2050?\",\n",
    "    \"What GDP growth and oil price does the Counterfactual Baseline assume?\",\n",
    "    \"Which energy sources are consumed most in 2050?\",\n",
    "    \"How large could U.S. liquefied natural gas exports become by 2050?\",\n",
    "    \"What happens to nuclear generation in the Low Oil and Gas Supply case?\",\n",
    "    \"How did past Annual Energy Outlook projections of total energy consumption compare with what actually happened?\",\n",
    "    \"What are the characteristics of trustworthy AI?\",\n",
    "    \"What are the four functions of the AI RMF Core?\",\n",
    "    \"What kinds of harm can AI systems cause?\",\n",
    "    \"How do AI risks differ from traditional software risks?\",\n",
    "    \"Who are the AI actors across the AI lifecycle?\",\n",
    "    \"What does the MEASURE function cover?\",\n",
    "]\n",
    "TOP_K = 3\n",
    "GEN_MODEL = \"gemini-2.5-flash\"\n",
    "MODEL_ID, MODEL_REVISION = \"vidore/colqwen2-v1.0-merged\", \"14ed9f04a49d1aa8ec08f3aad455a54eea9ee09a\"  # March 2025\n",
    "OUT = os.path.abspath(os.environ.get(\"OUT\", \"colpali_export\"))\n",
    "MAX_PAGES = int(os.environ.get(\"MAX_PAGES\", \"0\"))\n",
    "MAX_QUESTIONS = int(os.environ.get(\"MAX_QUESTIONS\", \"0\"))\n",
    "\n",
    "\n",
    "def step(title):\n",
    "    print(f\"\\n=== {title} ===\", flush=True)\n",
    "\n",
    "\n",
    "def main():\n",
    "    key = os.environ.get(\"GEMINI_API_KEY\") or None\n",
    "    for d in (\"pages\", \"maps\", \"pdfs\"):\n",
    "        os.makedirs(os.path.join(OUT, d), exist_ok=True)\n",
    "\n",
    "    step(\"1. Documents: the chosen pages go into the user's upload folder\")\n",
    "    settings.create_user_directories(USER_ID)\n",
    "    raw_dir = settings.RAW_DATA_DIR(USER_ID)\n",
    "    for f in os.listdir(raw_dir):\n",
    "        os.remove(os.path.join(raw_dir, f))\n",
    "    docs = DOCS\n",
    "    if MAX_PAGES:  # testing only: a few pages of the first document\n",
    "        docs = [dict(DOCS[0], pages=DOCS[0][\"pages\"][:MAX_PAGES])]\n",
    "    for d in docs:\n",
    "        req = urllib.request.Request(d[\"url\"], headers={\"User-Agent\": \"Mozilla/5.0\"})\n",
    "        reader, writer = PdfReader(io.BytesIO(urllib.request.urlopen(req, timeout=120).read())), PdfWriter()\n",
    "        for n in d[\"pages\"]:\n",
    "            writer.add_page(reader.pages[n - 1])\n",
    "        with open(os.path.join(raw_dir, d[\"file\"]), \"wb\") as fh:\n",
    "            writer.write(fh)\n",
    "        shutil.copy(os.path.join(raw_dir, d[\"file\"]), os.path.join(OUT, \"pdfs\", d[\"file\"]))\n",
    "        print(f\"{d['file']}: {len(d['pages'])} pages\", flush=True)\n",
    "\n",
    "    step(\"2. Model (the application's initialize_base_model)\")\n",
    "    model_path = os.environ.get(\"MODEL_PATH\")\n",
    "    if not model_path:\n",
    "        from huggingface_hub import snapshot_download\n",
    "        model_path = snapshot_download(MODEL_ID, revision=MODEL_REVISION)  # adjustment 1\n",
    "    initializers.COLQWEN2_MODEL_PATH = model_path\n",
    "    device = \"cuda\" if torch.cuda.is_available() else None  # adjustment 3\n",
    "    t = time.time()\n",
    "    model, processor = initializers.initialize_base_model(device_map=device)\n",
    "    device_name = torch.cuda.get_device_name() if device else \"CPU\"\n",
    "    print(f\"loaded in {time.time() - t:.0f} s on {device_name}\", flush=True)\n",
    "\n",
    "    step(\"3. Indexing (PDFProcessor.process_directory + EmbeddingProcessor.embed_documents)\")\n",
    "    images, pdf_sources = PDFProcessor.process_directory(USER_ID)\n",
    "    embedding_processor = EmbeddingProcessor(model, processor)\n",
    "    t = time.time()\n",
    "    document_embeddings, images, document_token_ids = embedding_processor.embed_documents(images)\n",
    "    index_seconds = time.time() - t\n",
    "    print(f\"indexed {len(images)} pages in {index_seconds:.1f} s ({index_seconds / len(images):.2f} s per page); \"\n",
    "          f\"{int(document_embeddings[0].shape[0])} vectors of {int(document_embeddings[0].shape[1])} numbers for the first page\", flush=True)\n",
    "\n",
    "    step(\"4. Questions (QueryProcessor.process_query: scores, top pages, Gemini answers, heat maps)\")\n",
    "    import google.generativeai as genai\n",
    "    gemini_model = None\n",
    "    if key:\n",
    "        initializers.initialize_gemini(api_key=key)  # the application's setup: configures the key\n",
    "        gemini_model = genai.GenerativeModel(  # adjustment 2: same settings, current model\n",
    "            model_name=GEN_MODEL, generation_config=genai.types.GenerationConfig(**settings.GEMINI_CONFIG))\n",
    "    else:\n",
    "        print(\"No GEMINI_API_KEY: retrieval and heat maps still run; answers are left empty.\", flush=True)\n",
    "    query_processor = QueryProcessor(embedding_processor, gemini_model, document_embeddings, images, pdf_sources, document_token_ids)\n",
    "    original_answer = query_processor._get_gemini_response\n",
    "\n",
    "    def answer_or_note(image, query):\n",
    "        if gemini_model is None:\n",
    "            return \"\"\n",
    "        try:\n",
    "            return original_answer(image, query)\n",
    "        except Exception as e:  # keep going: one failed call must not stop the run\n",
    "            print(f\"  Gemini call failed: {e}\", flush=True)\n",
    "            return \"\"\n",
    "    query_processor._get_gemini_response = answer_or_note\n",
    "\n",
    "    questions = QUESTIONS[:MAX_QUESTIONS] if MAX_QUESTIONS else QUESTIONS\n",
    "    results = []\n",
    "    for qi, q in enumerate(questions):\n",
    "        out = query_processor.process_query(q, top_k=min(TOP_K, len(images)), visualize=True, db=None, user_id=USER_ID)\n",
    "        q_emb = embedding_processor.embed_queries([q])[0]\n",
    "        scores = [float(embedding_processor.sum_max_sim(q_emb, e, ids)) for e, ids in zip(document_embeddings, document_token_ids)]\n",
    "        maps = []\n",
    "        for k, rel in enumerate(out.get(\"visualization_paths\") or [], start=1):\n",
    "            dst = f\"maps/q{qi + 1:02d}-match{k}.png\"\n",
    "            shutil.copy(os.path.join(settings.OUTPUT_DIR, rel), os.path.join(OUT, dst))\n",
    "            maps.append(dst)\n",
    "        results.append({\"question\": q, \"scores\": scores, \"matches\": out[\"matches\"], \"metrics\": out[\"metrics\"], \"maps\": maps})\n",
    "        print(f\"\\n{q}\", flush=True)\n",
    "        for m in out[\"matches\"]:\n",
    "            print(f\"  {m['source']} page {m['page']} (score {m['score']:.1f}): {(m['answer'] or '(no answer)').strip()[:160]}\", flush=True)\n",
    "\n",
    "    step(\"5. Saving the results for the website\")\n",
    "    pages = []\n",
    "    for i, (img, src) in enumerate(zip(images, pdf_sources)):\n",
    "        d = next(x for x in docs if x[\"file\"] == src[\"filename\"])\n",
    "        original_page = d[\"pages\"][src[\"page\"] - 1]\n",
    "        name = f\"pages/{d['id']}-{original_page:02d}.webp\"\n",
    "        small = img.convert(\"RGB\")\n",
    "        small.thumbnail((1100, 1100))\n",
    "        small.save(os.path.join(OUT, name), \"WEBP\", quality=82)\n",
    "        pages.append({\"index\": i, \"file\": src[\"filename\"], \"page_in_file\": src[\"page\"], \"doc\": d[\"id\"], \"page\": original_page, \"image\": name})\n",
    "    export = {\n",
    "        \"model\": MODEL_ID, \"model_revision\": MODEL_REVISION, \"generator\": GEN_MODEL if gemini_model else None,\n",
    "        \"gemini_config\": settings.GEMINI_CONFIG, \"top_k\": TOP_K, \"device\": device_name,\n",
    "        \"index_seconds\": round(index_seconds, 1), \"dim\": int(document_embeddings[0].shape[1]),\n",
    "        \"vectors_per_page\": [int(e.shape[0]) for e in document_embeddings],\n",
    "        \"docs\": [{k: d[k] for k in (\"id\", \"file\", \"title\", \"publisher\", \"url\", \"pages\")} for d in docs],\n",
    "        \"pages\": pages, \"questions\": results,\n",
    "    }\n",
    "    with open(os.path.join(OUT, \"colpali-demo.json\"), \"w\") as fh:\n",
    "        json.dump(export, fh, indent=1)\n",
    "    zip_path = OUT.rstrip(\"/\") + \".zip\"\n",
    "    with zipfile.ZipFile(zip_path, \"w\", zipfile.ZIP_DEFLATED) as z:\n",
    "        for root, _, files in os.walk(OUT):\n",
    "            for f in files:\n",
    "                z.write(os.path.join(root, f), os.path.relpath(os.path.join(root, f), OUT))\n",
    "    answered = sum(1 for r in results for m in r[\"matches\"] if m.get(\"answer\"))\n",
    "    print(f\"{zip_path}: {os.path.getsize(zip_path) / 1e6:.1f} MB; {len(pages)} pages, {len(results)} questions, {answered} answers\", flush=True)\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "bNXtztA6wYab"
   },
   "execution_count": 5,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "/content/env/lib/python3.11/site-packages/google/api_core/_python_version_support.py:254: FutureWarning: You are using a Python version (3.11.18) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2027-10-24). Please upgrade to the latest Python version, or at least Python 3.12, to continue receiving updates for google.api_core past that date.\n",
      "  warnings.warn(message, FutureWarning)\n",
      "\n",
      "=== 1. Documents: the chosen pages go into the user's upload folder ===\n",
      "nist-ai-rmf-selected-pages.pdf: 19 pages\n",
      "eia-aeo2026-slides-2-15.pdf: 14 pages\n",
      "\n",
      "=== 2. Model (the application's initialize_base_model) ===\n",
      "\n",
      "Fetching 12 files:   0%|          | 0/12 [00:00<?, ?it/s]\n",
      "Fetching 12 files:   8%|\u258a         | 1/12 [00:00<00:08,  1.36it/s]\n",
      "Fetching 12 files:  42%|\u2588\u2588\u2588\u2588\u258f     | 5/12 [00:00<00:01,  6.57it/s]\n",
      "Fetching 12 files:  42%|\u2588\u2588\u2588\u2588\u258f     | 5/12 [00:19<00:01,  6.57it/s]\n",
      "Fetching 12 files:  50%|\u2588\u2588\u2588\u2588\u2588     | 6/12 [00:30<00:42,  7.05s/it]\n",
      "Fetching 12 files: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 12/12 [00:30<00:00,  2.56s/it]\n",
      "`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46\n",
      "loaded in 3 s on NVIDIA L4\n",
      "\n",
      "=== 3. Indexing (PDFProcessor.process_directory + EmbeddingProcessor.embed_documents) ===\n",
      "INFO: Starting document embedding for 33 images\n",
      "INFO: Processing 17 batches\n",
      "\n",
      "Processing documents:   0%|          | 0/17 [00:00<?, ?it/s]\n",
      "Processing documents:   6%|\u258c         | 1/17 [00:04<01:09,  4.36s/it]\n",
      "Processing documents:  12%|\u2588\u258f        | 2/17 [00:08<01:01,  4.13s/it]\n",
      "Processing documents:  18%|\u2588\u258a        | 3/17 [00:12<00:56,  4.05s/it]\n",
      "Processing documents:  24%|\u2588\u2588\u258e       | 4/17 [00:16<00:52,  4.01s/it]\n",
      "Processing documents:  29%|\u2588\u2588\u2589       | 5/17 [00:20<00:47,  3.99s/it]\n",
      "Processing documents:  35%|\u2588\u2588\u2588\u258c      | 6/17 [00:24<00:43,  3.98s/it]\n",
      "Processing documents:  41%|\u2588\u2588\u2588\u2588      | 7/17 [00:28<00:39,  3.97s/it]\n",
      "Processing documents:  47%|\u2588\u2588\u2588\u2588\u258b     | 8/17 [00:32<00:36,  4.06s/it]\n",
      "Processing documents:  53%|\u2588\u2588\u2588\u2588\u2588\u258e    | 9/17 [00:36<00:32,  4.12s/it]\n",
      "Processing documents:  59%|\u2588\u2588\u2588\u2588\u2588\u2589    | 10/17 [00:40<00:29,  4.16s/it]\n",
      "Processing documents:  65%|\u2588\u2588\u2588\u2588\u2588\u2588\u258d   | 11/17 [00:45<00:25,  4.19s/it]\n",
      "Processing documents:  71%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588   | 12/17 [00:49<00:21,  4.21s/it]\n",
      "Processing documents:  76%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b  | 13/17 [00:53<00:16,  4.22s/it]\n",
      "Processing documents:  82%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f | 14/17 [00:57<00:12,  4.23s/it]\n",
      "Processing documents:  88%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a | 15/17 [01:02<00:08,  4.24s/it]\n",
      "Processing documents:  94%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d| 16/17 [01:06<00:04,  4.25s/it]\n",
      "Processing documents: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 17/17 [01:07<00:00,  3.34s/it]\n",
      "Processing documents: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 17/17 [01:07<00:00,  3.98s/it]\n",
      "INFO: Successfully embedded 33 documents\n",
      "indexed 33 pages in 67.6 s (2.05 s per page); 731 vectors of 128 numbers for the first page\n",
      "\n",
      "=== 4. Questions (QueryProcessor.process_query: scores, top pages, Gemini answers, heat maps) ===\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'Which end-use sector's energy consumption grows the most by 2050?' for user 1\n",
      "INFO: Found top 3 matches with scores: [21.126609802246094, 17.4249324798584, 16.080581665039062]\n",
      "INFO: Query processing completed in 32.47s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "Which end-use sector's energy consumption grows the most by 2050?\n",
      "  eia-aeo2026-slides-2-15.pdf page 7 (score 21.1): Based on the chart, the **industrial** sector's energy consumption grows the most by 2050.\n",
      "\n",
      "The green line, representing industrial energy consumption, shows a \n",
      "  eia-aeo2026-slides-2-15.pdf page 6 (score 17.4): Based on the provided image, there is no information about \"end-use sectors\" (such as residential, commercial, industrial, or transportation). Both charts displ\n",
      "  eia-aeo2026-slides-2-15.pdf page 10 (score 16.1): Based on the title of the image: \"Data center load is emerging as the dominant driver of long-term U.S. electricity growth,\" and the fact that the middle and ri\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'How much electricity could data center servers use in 2050 in the High Electricity Demand case?' for user 1\n",
      "INFO: Found top 3 matches with scores: [22.390151977539062, 20.769136428833008, 16.863985061645508]\n",
      "INFO: Query processing completed in 28.46s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "How much electricity could data center servers use in 2050 in the High Electricity Demand case?\n",
      "  eia-aeo2026-slides-2-15.pdf page 10 (score 22.4): Based on the middle chart, \"Commercial data center server electricity consumption\":\n",
      "\n",
      "In the **High Electricity Demand** case (the red line), the electricity con\n",
      "  eia-aeo2026-slides-2-15.pdf page 5 (score 20.8): Based on the provided image, the \"High Electricity Demand\" case **examines uncertainty about long-term computational requirements and data center server power d\n",
      "  eia-aeo2026-slides-2-15.pdf page 11 (score 16.9): Based on the provided image, the information about electricity usage specifically for \"data center servers\" in any case, including the \"High Electricity Demand\"\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'What share of light-duty vehicles could be battery electric by 2050?' for user 1\n",
      "INFO: Found top 3 matches with scores: [18.4947452545166, 16.946258544921875, 14.282910346984863]\n",
      "INFO: Query processing completed in 31.73s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "What share of light-duty vehicles could be battery electric by 2050?\n",
      "  eia-aeo2026-slides-2-15.pdf page 9 (score 18.5): Based on the \"Battery electric vehicle share of light-duty vehicle stocks\" chart (the left graph), the data provided only extends to **2045**, not 2050.\n",
      "\n",
      "Howeve\n",
      "  eia-aeo2026-slides-2-15.pdf page 8 (score 16.9): Based on the provided image, the share of light-duty vehicles that could be battery electric by 2050 is **not explicitly stated or shown**.\n",
      "\n",
      "The left chart (\"U.\n",
      "  eia-aeo2026-slides-2-15.pdf page 5 (score 14.3): I'm sorry, but the provided image does not contain information about the share of light-duty vehicles that could be battery electric by 2050.\n",
      "\n",
      "The image describ\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'What GDP growth and oil price does the Counterfactual Baseline assume?' for user 1\n",
      "INFO: Found top 3 matches with scores: [18.805736541748047, 16.49365997314453, 15.897980690002441]\n",
      "INFO: Query processing completed in 29.53s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "What GDP growth and oil price does the Counterfactual Baseline assume?\n",
      "  eia-aeo2026-slides-2-15.pdf page 4 (score 18.8): The Counterfactual Baseline assumes:\n",
      "*   **GDP Growth:** 1.7% annual GDP growth\n",
      "*   **Oil Price:** Brent crude at $87 per barrel in 2050\n",
      "  eia-aeo2026-slides-2-15.pdf page 13 (score 16.5): The provided image does not contain information about the GDP growth or oil price assumptions for the \"Counterfactual Baseline\" scenario.\n",
      "\n",
      "The image shows proje\n",
      "  eia-aeo2026-slides-2-15.pdf page 11 (score 15.9): The provided image **does not explicitly state** the specific GDP growth and oil price assumptions for the \"Counterfactual Baseline\" scenario.\n",
      "\n",
      "In EIA's Annual \n",
      "INFO: QueryProcessor: Starting process_query with query text: 'Which energy sources are consumed most in 2050?' for user 1\n",
      "INFO: Found top 3 matches with scores: [17.21845245361328, 15.050333023071289, 14.821001052856445]\n",
      "INFO: Query processing completed in 29.29s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "Which energy sources are consumed most in 2050?\n",
      "  eia-aeo2026-slides-2-15.pdf page 6 (score 17.2): According to the title of the image and both charts, **Natural gas** and **Petroleum and other liquids** are the energy sources consumed most in 2050.\n",
      "\n",
      "The righ\n",
      "  eia-aeo2026-slides-2-15.pdf page 7 (score 15.1): In 2050, based on the projections in the graph, the **industrial** sector is projected to consume the most energy.\n",
      "  eia-aeo2026-slides-2-15.pdf page 11 (score 14.8): Based on the \"Total generation in all sectors, 2025 and 2050\" bar chart (the right side of the image, looking at the rightmost part of each bar for 2050):\n",
      "\n",
      "The \n",
      "INFO: QueryProcessor: Starting process_query with query text: 'How large could U.S. liquefied natural gas exports become by 2050?' for user 1\n",
      "INFO: Found top 3 matches with scores: [22.72726821899414, 18.87939453125, 16.117679595947266]\n",
      "INFO: Query processing completed in 28.58s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "How large could U.S. liquefied natural gas exports become by 2050?\n",
      "  eia-aeo2026-slides-2-15.pdf page 12 (score 22.7): Looking at the \"U.S. liquefied natural gas exports\" chart (the rightmost one):\n",
      "\n",
      "The highest projection for U.S. liquefied natural gas exports by 2050 is represe\n",
      "  eia-aeo2026-slides-2-15.pdf page 13 (score 18.9): Based on the provided image, it is not possible to determine the potential size of U.S. liquefied natural gas (LNG) exports by 2050.\n",
      "\n",
      "The charts in the image di\n",
      "  eia-aeo2026-slides-2-15.pdf page 6 (score 16.1): I'm sorry, but the provided image focuses on **U.S. energy consumption by fuel type** under various scenarios for 2025 and 2050 (and 2030, 2040 for the right ch\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'What happens to nuclear generation in the Low Oil and Gas Supply case?' for user 1\n",
      "INFO: Found top 3 matches with scores: [17.23406982421875, 15.533129692077637, 15.48381519317627]\n",
      "INFO: Query processing completed in 33.78s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "What happens to nuclear generation in the Low Oil and Gas Supply case?\n",
      "  eia-aeo2026-slides-2-15.pdf page 14 (score 17.2): In the \"Low Oil and Gas Supply\" case (represented by the **yellow line**), U.S. nuclear electricity generation is projected to **increase significantly** after \n",
      "  eia-aeo2026-slides-2-15.pdf page 11 (score 15.5): In the \"Low Oil and Gas Supply\" case, nuclear generation (represented by the dark red segment) appears to remain largely unchanged in absolute terms compared to\n",
      "  eia-aeo2026-slides-2-15.pdf page 6 (score 15.5): In the **Low Oil and Gas Supply case**, nuclear consumption (represented by the dark red/maroon segment) remains relatively stable between 2025 and 2050.\n",
      "\n",
      "*   I\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'How did past Annual Energy Outlook projections of total energy consumption compare with what actually happened?' for user 1\n",
      "INFO: Found top 3 matches with scores: [17.960603713989258, 17.93341064453125, 16.627437591552734]\n",
      "INFO: Query processing completed in 38.12s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "How did past Annual Energy Outlook projections of total energy consumption compare with what actually happened?\n",
      "  eia-aeo2026-slides-2-15.pdf page 2 (score 18.0): The image states that the EIA (U.S. Energy Information Administration) **does** compare Annual Energy Outlook (AEO) projections to what actually happened.\n",
      "\n",
      "Spec\n",
      "  eia-aeo2026-slides-2-15.pdf page 3 (score 17.9): Based on the \"Total energy consumption from all sectors\" chart (left side of the image):\n",
      "\n",
      "Past Annual Energy Outlook (AEO) projections of total energy consumpti\n",
      "  eia-aeo2026-slides-2-15.pdf page 7 (score 16.6): Based on the provided image, it is **not possible to directly compare past Annual Energy Outlook (AEO) projections with what actually happened.**\n",
      "\n",
      "Here's why:\n",
      "\n",
      "\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'What are the characteristics of trustworthy AI?' for user 1\n",
      "INFO: Found top 3 matches with scores: [14.696130752563477, 14.358141899108887, 13.47156047821045]\n",
      "INFO: Query processing completed in 34.14s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "What are the characteristics of trustworthy AI?\n",
      "  nist-ai-rmf-selected-pages.pdf page 8 (score 14.7): Based on the provided image, the characteristics of trustworthy AI systems include:\n",
      "\n",
      "*   **Valid and Reliable**\n",
      "*   **Safe**\n",
      "*   **Secure and Resilient**\n",
      "*   **\n",
      "  nist-ai-rmf-selected-pages.pdf page 15 (score 14.4): Based on the provided image, the document **mentions** the \"characteristics of trustworthy AI\" but **does not explicitly list or define them** within this parti\n",
      "  nist-ai-rmf-selected-pages.pdf page 10 (score 13.5): Based on the provided image, the characteristics of trustworthy AI are:\n",
      "\n",
      "1.  **Safety**: Employing safety considerations during the lifecycle to prevent failure\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'What are the four functions of the AI RMF Core?' for user 1\n",
      "INFO: Found top 3 matches with scores: [17.484535217285156, 14.83258056640625, 14.143410682678223]\n",
      "INFO: Query processing completed in 31.75s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "What are the four functions of the AI RMF Core?\n",
      "  nist-ai-rmf-selected-pages.pdf page 14 (score 17.5): According to the text, the four functions of the AI RMF Core are:\n",
      "\n",
      "1.  **GOVERN**\n",
      "2.  **MAP**\n",
      "3.  **MEASURE**\n",
      "4.  **MANAGE**\n",
      "  nist-ai-rmf-selected-pages.pdf page 3 (score 14.8): The four functions of the AI RMF Core are:\n",
      "\n",
      "1.  **GOVERN**\n",
      "2.  **MAP**\n",
      "3.  **MEASURE**\n",
      "4.  **MANAGE**\n",
      "  nist-ai-rmf-selected-pages.pdf page 6 (score 14.1): Based on the provided image and text, the four functions of the AI RMF Core are **not explicitly listed**.\n",
      "\n",
      "The text states: \"The AI RMF functions, described in\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'What kinds of harm can AI systems cause?' for user 1\n",
      "INFO: Found top 3 matches with scores: [15.028825759887695, 14.803064346313477, 13.978117942810059]\n",
      "INFO: Query processing completed in 33.61s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "What kinds of harm can AI systems cause?\n",
      "  nist-ai-rmf-selected-pages.pdf page 5 (score 15.0): Based on the provided image (Fig. 1), AI systems can cause potential harms in three main categories:\n",
      "\n",
      "1.  **Harm to People:**\n",
      "    *   **Individual:** Harm to a \n",
      "  nist-ai-rmf-selected-pages.pdf page 4 (score 14.8): According to the provided text, AI systems can cause various kinds of harm, generally referred to as \"potential negative impacts,\" \"threats,\" or \"harms.\"\n",
      "\n",
      "Speci\n",
      "  nist-ai-rmf-selected-pages.pdf page 9 (score 14.0): Based on the provided text, AI systems can cause harm that endangers:\n",
      "\n",
      "*   **Human life** (specifically mentioned as \"serious injury or death\" in the context of\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'How do AI risks differ from traditional software risks?' for user 1\n",
      "INFO: Found top 3 matches with scores: [17.508827209472656, 15.160322189331055, 14.054533004760742]\n",
      "INFO: Query processing completed in 48.38s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "How do AI risks differ from traditional software risks?\n",
      "  nist-ai-rmf-selected-pages.pdf page 19 (score 17.5): According to the provided text, AI risks differ from traditional software risks in several key ways, often being broader, more complex, and not fully addressed \n",
      "  nist-ai-rmf-selected-pages.pdf page 1 (score 15.2): Based on the provided text, AI risks differ from traditional software risks in several unique ways:\n",
      "\n",
      "1.  **Dynamic Data Dependency:** AI systems are \"trained on\n",
      "  nist-ai-rmf-selected-pages.pdf page 8 (score 14.1): Based on the provided text, AI risks differ from traditional software risks primarily in their **scope, nature, and the complexity of managing them**, especiall\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'Who are the AI actors across the AI lifecycle?' for user 1\n",
      "INFO: Found top 3 matches with scores: [13.253778457641602, 13.18555736541748, 12.761420249938965]\n",
      "INFO: Query processing completed in 41.33s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "Who are the AI actors across the AI lifecycle?\n",
      "  nist-ai-rmf-selected-pages.pdf page 6 (score 13.3): Based on the provided image and text:\n",
      "\n",
      "1.  The diagram (Figure 2) itself does not explicitly list the \"AI actors.\" Its central core, **\"People and Planet,\"** br\n",
      "  nist-ai-rmf-selected-pages.pdf page 2 (score 13.2): According to the provided text, **AI actors** are defined by the Organisation for Economic Co-operation and Development (OECD) as:\n",
      "\n",
      "\"**those who play an active \n",
      "  nist-ai-rmf-selected-pages.pdf page 7 (score 12.8): Based on the \"Representative Actors\" column in the provided image (Fig. 3. AI actors across AI lifecycle stages), the AI actors across the AI lifecycle include:\n",
      "INFO: QueryProcessor: Starting process_query with query text: 'What does the MEASURE function cover?' for user 1\n",
      "INFO: Found top 3 matches with scores: [15.201284408569336, 12.675163269042969, 11.935125350952148]\n",
      "INFO: Query processing completed in 31.80s\n",
      "INFO: Visualization for match 1 saved to: user_1/match_1.png\n",
      "INFO: Visualization for match 2 saved to: user_1/match_2.png\n",
      "INFO: Visualization for match 3 saved to: user_1/match_3.png\n",
      "\n",
      "What does the MEASURE function cover?\n",
      "  nist-ai-rmf-selected-pages.pdf page 17 (score 15.2): The MEASURE function, as described in the provided text and Table 3, covers practices related to **measuring AI risks** and **evaluating AI systems for trustwor\n",
      "  nist-ai-rmf-selected-pages.pdf page 3 (score 12.7): Based on the provided text, the **MEASURE** function is one of four specific functions (along with GOVERN, MAP, and MANAGE) that comprise the \"Core\" of the Fram\n",
      "  nist-ai-rmf-selected-pages.pdf page 14 (score 11.9): Based on the provided image, the MEASURE function covers:\n",
      "\n",
      "*   **Identified risks are assessed, analyzed, or tracked.**\n",
      "\n",
      "=== 5. Saving the results for the website ===\n",
      "/content/colpali_export.zip: 58.7 MB; 33 pages, 14 questions, 42 answers\n"
     ]
    }
   ],
   "source": [
    "import os, subprocess\n",
    "env = {k: v for k, v in os.environ.items() if k not in (\"PYTHONPATH\", \"MPLBACKEND\")}\n",
    "env.update(MPLBACKEND=\"Agg\", OUT=\"/content/colpali_export\", PYTHONUNBUFFERED=\"1\")\n",
    "try:\n",
    "    from google.colab import userdata\n",
    "    env[\"GEMINI_API_KEY\"] = userdata.get(\"GEMINI_API_KEY\")\n",
    "except Exception as e:\n",
    "    print(\"No Gemini key available (\", type(e).__name__, \"): the answers will be left empty.\")\n",
    "proc = subprocess.Popen([\"/content/env/bin/python\", \"run_demo.py\"], cwd=\"/content/app\", env=env,\n",
    "                        stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)\n",
    "for line in proc.stdout:\n",
    "    print(line, end=\"\")\n",
    "if proc.wait() != 0:\n",
    "    raise SystemExit(\"The run stopped with an error (see above).\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4ZjBD7kiwYab"
   },
   "source": [
    "## 4. Download the results"
   ],
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "XzRmvnfvwYac"
   },
   "execution_count": 6,
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "files.download(\"/content/colpali_export.zip\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": [],
   "machine_shape": "hm",
   "gpuType": "L4"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}