{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9e8b63ab",
   "metadata": {},
   "source": [
    "# Build Your First Multimodal AI Application — Beginner's Notebook (2026)\n",
    "\n",
    "**AI Business Help** | [Read the accompanying tutorial](https://aibusinesshelp.co.uk/how-to-build-a-multimodal-ai-model-step-by-step-tutorial-for-beginners)\n",
    "\n",
    "This free notebook contains three exercises:\n",
    "\n",
    "1. **CLIP:** Match an image to written descriptions.\n",
    "2. **SmolVLM:** Ask an AI to describe an image.\n",
    "3. **Your own photo:** Upload an image and ask a question about it.\n",
    "\n",
    "**How to use:** Open the `.ipynb` file in [Google Colab](https://colab.research.google.com/), then click **Runtime → Run all**, or run each code cell with the ▶ icon. An internet connection is needed to install packages and download the model weights. The notebook does not train models or update their weights.\n",
    "\n",
    "**Hardware:** Start with a CPU runtime. A GPU may make the second experiment faster, but free GPU availability is not guaranteed. The first run downloads model files, which can take time.\n",
    "\n",
    "**Privacy:** Don't upload sensitive or personal images unless you understand and accept the cloud environment's data-handling arrangements."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b2565f2",
   "metadata": {},
   "source": [
    "## Step 1 — Install Python packages\n",
    "Run this cell first. If Colab asks you to restart the runtime after installing packages, restart it and run from Step 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd48698e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip -q install -U \"transformers>=4.52,<6\" torch pillow requests accelerate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b7be66",
   "metadata": {},
   "source": [
    "## Step 2 — Download a sample image\n",
    "We will use a public demonstration image of a bee on a flower."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21c1c3db",
   "metadata": {},
   "outputs": [],
   "source": [
    "from io import BytesIO\n",
    "import requests\n",
    "from PIL import Image\n",
    "from IPython.display import display\n",
    "\n",
    "IMAGE_URL = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\"\n",
    "response = requests.get(IMAGE_URL, timeout=45)\n",
    "response.raise_for_status()\n",
    "image = Image.open(BytesIO(response.content)).convert(\"RGB\")\n",
    "# Keep the demo small enough for an introductory notebook\n",
    "image.thumbnail((800, 800))\n",
    "display(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ca3655b",
   "metadata": {},
   "source": [
    "## Experiment 1 — Compare an image with text using CLIP\n",
    "\n",
    "CLIP assigns each description a **relative matching score**. The scores below are not proof that any caption is true; the supplied alternatives influence the percentages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7842917",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import CLIPModel, CLIPProcessor\n",
    "\n",
    "clip_id = \"openai/clip-vit-base-patch32\"\n",
    "clip_model = CLIPModel.from_pretrained(clip_id)\n",
    "clip_processor = CLIPProcessor.from_pretrained(clip_id)\n",
    "clip_model.eval()\n",
    "\n",
    "descriptions = [\n",
    "    \"A bee sitting on a flower\",\n",
    "    \"A dog running through a field\",\n",
    "    \"A car parked on a road\",\n",
    "    \"A person riding a bicycle\",\n",
    "]\n",
    "\n",
    "inputs = clip_processor(\n",
    "    text=descriptions, images=image, return_tensors=\"pt\", padding=True\n",
    ")\n",
    "with torch.inference_mode():\n",
    "    output = clip_model(**inputs)\n",
    "    scores = output.logits_per_image.softmax(dim=1)[0]\n",
    "\n",
    "for caption, score in sorted(zip(descriptions, scores.tolist()), key=lambda item: item[1], reverse=True):\n",
    "    print(f\"{score:6.1%}  {caption}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74800945",
   "metadata": {},
   "source": [
    "**Try it:** Change the descriptions and rerun the CLIP cell. What happens when none of your descriptions is accurate?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "535b9e7f",
   "metadata": {},
   "source": [
    "## Experiment 2 — Ask a vision-language model about the image\n",
    "\n",
    "We'll load **SmolVLM-256M-Instruct**, a compact image-to-text model. It generates answers in words instead of comparing a predefined list of captions.\n",
    "\n",
    "The model can still make mistakes or invent details. Treat generated descriptions as suggestions, not verified facts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2423a96a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "vlm_id = \"HuggingFaceTB/SmolVLM-256M-Instruct\"\n",
    "# Automatically choose GPU when available, otherwise CPU.\n",
    "vlm = pipeline(\"image-text-to-text\", model=vlm_id, device=0 if torch.cuda.is_available() else -1)\n",
    "print(\"Vision-language model ready on\", \"GPU\" if torch.cuda.is_available() else \"CPU\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8fece34",
   "metadata": {},
   "outputs": [],
   "source": [
    "def ask_about_image(picture, question, max_new_tokens=100):\n",
    "    messages = [{\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\"type\": \"image\", \"image\": picture},\n",
    "            {\"type\": \"text\", \"text\": question},\n",
    "        ],\n",
    "    }]\n",
    "    results = vlm(text=messages, max_new_tokens=max_new_tokens, return_full_text=False)\n",
    "    answer = results[0][\"generated_text\"]\n",
    "    # On some library versions, generated_text may be a structured list.\n",
    "    if isinstance(answer, list):\n",
    "        answer = \" \".join(part.get(\"text\", \"\") for part in answer if isinstance(part, dict))\n",
    "    return str(answer).strip()\n",
    "\n",
    "print(ask_about_image(image, \"Describe what you can see in this image.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1666707f",
   "metadata": {},
   "source": [
    "**Try it:** Change the question to `What colours are visible?` or `Write a short alt text for this image.`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c994bf0",
   "metadata": {},
   "source": [
    "## Experiment 3 — Upload your own photograph (optional)\n",
    "\n",
    "The upload dialog works in **Google Colab**. Select a `.jpg`, `.jpeg`, `.png`, or `.webp` image. The notebook uses only the first uploaded file. Uploaded files are temporary in the Colab runtime.\n",
    "\n",
    "**Only upload images you are comfortable processing in Colab.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a38dadd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "\n",
    "uploaded = files.upload()\n",
    "if not uploaded:\n",
    "    print(\"No file selected. Rerun this cell when you're ready.\")\n",
    "else:\n",
    "    filename = next(iter(uploaded))\n",
    "    with Image.open(BytesIO(uploaded[filename])) as uploaded_image:\n",
    "        my_image = uploaded_image.convert(\"RGB\")\n",
    "    my_image.thumbnail((800, 800))\n",
    "    display(my_image)\n",
    "    print(\"Uploaded:\", filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaf9cf47",
   "metadata": {},
   "outputs": [],
   "source": [
    "if \"my_image\" in globals():\n",
    "    question = \"Describe the main objects in this photograph.\"\n",
    "    print(\"Question:\", question)\n",
    "    print(\"Answer:\", ask_about_image(my_image, question, max_new_tokens=120))\n",
    "else:\n",
    "    print(\"Upload an image in the previous cell first.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d621cb36",
   "metadata": {},
   "source": [
    "## What have you built?\n",
    "\n",
    "- **Experiment 1:** A program that compares an image with candidate captions (CLIP).\n",
    "- **Experiment 2:** A program that generates a description or answer about an image (SmolVLM).\n",
    "- **Experiment 3:** The same application using your own image.\n",
    "\n",
    "These examples run **pretrained models**. They do *not* train or fine-tune the models. To fine-tune a model you need an appropriate dataset, a supported training setup and a separate evaluation process.\n",
    "\n",
    "### Troubleshooting\n",
    "\n",
    "- **Package/import error?** Restart the runtime after installation and run cells again from the top.\n",
    "- **Out of memory?** Restart the runtime, use smaller images, or switch to a suitable GPU runtime if available.\n",
    "- **Slow second experiment?** CPU inference can be slow, and model download speeds depend on connectivity.\n",
    "- **A strange answer?** Small vision-language models can hallucinate. Try a simpler question and check the original image.\n",
    "- **Upload fails locally?** The upload step is designed for Colab. For local Python, replace it with `Image.open(\"/path/to/photo.jpg\").convert(\"RGB\")`.\n",
    "\n",
    "### References\n",
    "\n",
    "- [Hugging Face image-text-to-text documentation](https://huggingface.co/docs/transformers/tasks/image_text_to_text)\n",
    "- [Hugging Face CLIP documentation](https://huggingface.co/docs/transformers/model_doc/clip)\n",
    "- [SmolVLM model page](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct)\n",
    "- [Google Colab FAQ](https://research.google.com/colaboratory/faq.html)\n",
    "\n",
    "**AI Business Help:** [Back to the full tutorial](https://aibusinesshelp.co.uk/how-to-build-a-multimodal-ai-model-step-by-step-tutorial-for-beginners)\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "AI Business Help - Multimodal AI Beginners Tutorial.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
