Multi-Modal Prompting: Working with Images, Code, and Text in GPT-4 and Claude

        <h2>The Multi-Modal Shift</h2>
        <p>Modern LLMs understand more than text. GPT-4V, Claude 3.5, and Gemini Pro Vision can process images, diagrams, screenshots, and documents alongside text instructions. This fundamentally changes what prompts can do — you're no longer limited to describing things in words; you can <strong>show the model what you mean</strong>.</p>
        <p>Multi-modal prompting isn't just "upload an image and ask a question." The real power comes from combining modalities strategically to solve problems that neither text nor vision alone can handle.</p>

        <h2>Vision + Text: Core Patterns</h2>
        <h3>Pattern 1: Screenshot-to-Code</h3>
        <p>Convert UI screenshots or mockups directly into working code:</p>
        <pre><code>System: You are a senior frontend developer. Convert the provided UI screenshot 

        <h3>Pattern 2: Diagram Analysis</h3>
        <p>Extract structured information from diagrams, flowcharts, and architecture drawings:</p>
        <pre><code>System: Analyse the attached architecture diagram and extract the following:

        <h3>Pattern 3: Document Processing</h3>
        <p>Extract data from forms, receipts, invoices, and scanned documents:</p>
        <pre><code>System: Extract all data from this invoice image into a structured JSON format.

        <h2>Image + Code: Advanced Patterns</h2>
        <h3>Visual Regression Testing</h3>
        <p>Use vision models to compare UI states and identify visual regressions:</p>
        <pre><code>System: You are a QA engineer performing visual regression testing.

        <h3>Debug from Screenshots</h3>
        <p>When a user reports a bug with a screenshot, combine the visual with code context:</p>
        <pre><code>System: A user reported the following bug with the attached screenshot.

{component_code}

        <h2>Multi-Image Prompting</h2>
        <p>Sending multiple images in a single prompt enables comparison, sequence analysis, and richer context:</p>
        <ul>
            <li><strong>Before/After comparisons</strong> — Show two states and ask for a diff</li>
            <li><strong>Multi-page documents</strong> — Process entire PDF-like documents page by page</li>
            <li><strong>Design systems</strong> — Show multiple component examples to establish a pattern, then generate new components</li>
            <li><strong>Sequential UI flows</strong> — Show a user journey across screens and identify UX issues</li>
        </ul>

        <h2>Model Comparison for Vision Tasks</h2>
        <table>
            <tr><th>Task</th><th>Best Model</th><th>Notes</th></tr>
            <tr><td>Screenshot-to-code</td><td>Claude 3.5 Sonnet</td><td>Best at matching visual details; cleaner code output</td></tr>
            <tr><td>Document extraction</td><td>GPT-4 Vision</td><td>Strong OCR; handles messy handwriting better</td></tr>
            <tr><td>Diagram analysis</td><td>Gemini Pro Vision</td><td>Good spatial reasoning; handles complex diagrams well</td></tr>
            <tr><td>Visual regression</td><td>Claude 3.5 Sonnet</td><td>Most reliable at spotting subtle pixel differences</td></tr>
            <tr><td>Chart/graph reading</td><td>GPT-4 Vision</td><td>Best at extracting numerical data from charts</td></tr>
        </table>

        <h2>Cost Optimisation for Vision</h2>
        <p>Vision tokens are expensive. Key optimisation strategies:</p>
        <ul>
            <li><strong>Resize images</strong> — Most models scale images internally. Sending a 4K screenshot wastes tokens. Resize to 1024px or 768px width</li>
            <li><strong>Crop to region of interest</strong> — If you only care about a button, don't send the full page</li>
            <li><strong>Use low-detail mode</strong> — GPT-4V supports <code>detail: "low"</code> for tasks that don't need pixel-level precision</li>
            <li><strong>Cache results</strong> — If the same image is analysed repeatedly, cache the extracted data</li>
            <li><strong>Text over images when possible</strong> — If you can describe something in text, that's cheaper than an image</li>
        </ul>

        <h2>How AI Prompt Architect Helps</h2>
        <p>While AI Prompt Architect currently focuses on text-based prompt engineering, the prompt patterns generated by our <strong>Generate</strong> workflow are designed to be modality-aware. When you specify a vision-related task, the system scaffolds your prompt with appropriate image analysis instructions, output format specifications, and multi-modal best practices — saving you from the trial-and-error of crafting vision prompts from scratch.</p>