What I Could and Couldn't Verify Self-Hosting Open Models: Colab, RunPod, and an API Direction for Final Reviews

Summary

I started from a Qwen Gradio app that runs in Google Colab, then rented GPUs on RunPod to connect open-weight models to Open WebUI. That happened in three phases:

• Phase 1 (2026-10-09): a small Qwen2.5-1.5B-Instruct model returned one reply through Open WebUI.
• Phase 2 (2026-10-15): Qwen3.8-27B in BF16 returned one reply the same way. I took no quality or speed benchmarks.
• Phase 3 (2026-10-15): I tried to run a GGUF build of GLM-5.3-Flash on an H200 SXM. The container image pull completed, and I stopped the run while the model weights were still downloading. That produced zero model outputs and no A/B evaluation.

The same day, my published final retrospective ended the RunPod evaluation. For ordinary review of code, documents and final deliverables, it records the MiMo-V2.6-Pro API as an unverified operational candidate (no API account, key, call or result has been verified).

My own account today, which the public repository doesn't contain: ordinary code, prose and final-artifact review became the recurring use case, and my usage frequency and the operational burden of running GPUs led me to take the MiMo-V2.6-Pro API as my practical direction. I have not called it, integrated it, benchmarked it, or shown it to be cheaper or better. I also don't consider the self-hosting work wasted. It remains useful technical learning for me.

The rest of this post covers the setup, the order of the work, what the records actually support, the obstacles and how I triaged them, the work that never shows up on a GPU bill, the questions that separate an API from self-hosting, and where self-hosting still makes sense.

How to read the numbers

I separate four kinds of statement in the text: measured values from the records, estimates, my own decisions, and general inference.

Dollar figures are estimates. Each is a measurement window multiplied by the rate shown at the time. The window runs from just before Deploy until the $0/hr display after Stop. It is not response latency and not GPU-only time, and none of the figures is a final invoice. The Phase 1 figure covers only its final trial. Because the windows differ and no invoice has been reconciled, I don't add the phases into a total.

Links point to the public docs on main.

The setup

This is a simplified view of Phases 1 and 2. Phase 3 had the same pod-side shape with llama.cpp as the server, but it never got as far as a model response.

Open WebUI
   │  authenticated HTTPS, through RunPod's HTTPS proxy
   ▼
GPU pod on RunPod
   └─ inference server (Phases 1-2: vLLM / Phase 3: llama.cpp server)
        └─ open-weight model (Qwen, then a GLM-5.3-Flash GGUF)

• Phase 1: an authentication guard sat in front, and only GET /v1/models and POST /v1/chat/completions were allowed through. vLLM listened behind it on loopback. (Phase 1 record; this record is in Japanese only.)
• Phase 2: the vLLM API key was injected from a RunPod Secret reference, and a local Open WebUI used an OpenAI-compatible connection. An unauthenticated /v1/models request returned 401 before I sent anything. (Phase 2 record)
• Phase 3: a llama.cpp server image pinned by digest, told to fetch GGUF weights itself. (Phase 3 trial record)

For Phases 2 and 3, I set pre-approved limits before starting (Phase 2: 45 minutes and $3; Phase 3: 75 minutes, $7, and six stop conditions). In Phase 2 I finished the UI-side setup before the GPU started, then ran Deploy, one request, Stop, a $0 spend-rate check, and Terminate in one sequence. In Phase 3 I stopped and terminated under those limits.

I also designed a job API and runner that would save results before stopping the pod, but never built it (README). The RunPod lifecycle code lives in a separate repo, qwen-runpod-operations, according to the Phase 2 record. I haven't re-read that repo for this post.

The order of the work

1. A Qwen Gradio app on Colab. The repository began as a Qwen Gradio app and then tried on-demand RunPod inference (README). The README lists two Colab notebooks, one chat-only and one multimodal. The app is still in the repository, but I did not deploy and verify the whole app or its multimodal workflow on RunPod (final retrospective). The public records don't say why I moved to RunPod, so this post doesn't either.
2. Phase 1 (2026-10-09). A minimal connectivity test with a small model.
3. Phase 2 (2026-10-15, PR #44). The same path with a 27B-class model, as one short connection test.
4. Phase 3 (2026-10-15, PR #45). A planned comparison of three GLM variants (A: a GGUF of the official model; B: an abliterated GGUF variant; C: a security-tuned FP8 build based on full GLM-5.3 rather than Flash). I finished the GPU-free preparation and started track A. The README treats the whole plan as historical.
5. Closing (2026-10-15, PR #46). The final retrospective ended the evaluation and recorded the API direction for review work.

What the records actually support

Sources: Phase 1, Phase 2, Phase 3 trial, and the final retrospective. Each run ended with a $0/hr display after Stop and a $0 spend rate after the pod was deleted. The Phase 2 and 3 records add a pod count of 0, and Phase 3 also records the endpoint returning 404.

Two cautions about reading this table:

• Phases 1 and 2 are connection tests. "One reply" in Phase 2 means one reply came back through the documented A100 and Qwen3.8-27B BF16 path. It says nothing general about quality or speed.
• The Phase 3 preparation included a scorer that I checked against a mock server. Its scores are test-fixture values, not GLM measurements, and I don't use them here.

Obstacles and how I triaged them

The final retrospective is explicit that these were operational faults, distinct from reasoning or answer capability, and that platform or security restrictions shouldn't be read as model inability.

The GPU didn't come back after Stop (Phase 1, earlier trial)

An earlier trial got as far as a ready model but never confirmed a reply in Open WebUI. After I stopped it, I couldn't get a GPU back on the same pod. According to the record, RunPod releases the GPU on Stop, so a stopped pod can't resume if another user has taken the GPU on its original host. The container disk is also wiped on Stop. For the final trial I prepared the UI first and only updated the connection target once a fresh pod was up (Phase 1 record).

Stripped quotes and a restart loop (Phase 2)

I put an argument with a JSON value in the start command. RunPod stripped its quotes, vLLM exited on the argument error, and the pod restarted in a loop. /health returning 404 or 502 can't tell a restart loop from a slow load, so the lesson I recorded is to read the System and Container logs in the first minute. I fixed it by removing the argument and editing the pod. No weights had been fetched yet, so only time was lost (Phase 2 record).

The first request was rejected (Phases 1 and 2)

In Phase 2, Open WebUI 0.12.2 defaulted Function Calling to Native. It sent enabled tools with tool_choice: "auto", and vLLM rejected the request because the server had no tool-call settings. Switching the chat to Legacy and regenerating fixed it. The same switch worked in Phase 1, but there I never confirmed the rejection's cause from the actual payload.

A thinking model and a small max_tokens (Phase 2)

Qwen3.8 starts by thinking, so 64 tokens can run out before any answer appears. Sending chat_template_kwargs = {"enable_thinking": false} as an Open WebUI custom parameter returned only the answer.

Template defaults (Phase 2)

RunPod's "vLLM Latest" template defaulted to a guessable API key and to a HF_HOME that assumed a persistent volume. I replaced the first with a Secret reference and moved the second onto the container disk.

A /health that stayed at 502 (Phase 3)

In Phase 3 /health returned 502 the whole time. llama-server doesn't start listening until the -hf fetch finishes, so this wasn't an error signal, and there were no server errors or restart loops. I followed progress through container-disk usage (JST):

The measurement started at 23:08:35.5, and the image pull and container creation finished at 23:10:45. The image pull completed; I stopped the run while the weights were downloading. My stop condition 3 required readiness within 25 minutes of model start, and the projection at the time put fetch completion near 23:54 and readiness near 23:58. I gave the abort order at 23:31:12, the $0.00/hr display after Stop appeared at 23:31:37, and I terminated the pod after that (Phase 3 trial record). The record doesn't establish why the fetch slowed, and I won't guess.

The final retrospective frames Phase 3 as not an image-download failure, not evidence that the model is poor, and not evidence that RunPod is generally unusable. Phases 1 and 2 each reached one reply on RunPod.

GPU supply (Phase 3)

My planned H200 NVL went out of stock right after approval, and switching to the H200 SXM, at about 1.4 times the rate, needed a fresh approval. The record treats GPU supply as one more variable.

Work that never appears on the GPU bill

The public records show these non-GPU items:

• GPU-free preparation. Phase 3's P3-0 covered a model registry with pinned revisions (via Hugging Face's public API, with no downloads), a check of the first 8 MB of the GGUF headers, a check that the inference image contained the target model's implementation, an evaluation set, and a scorer validated against a mock server. The first version of the scorer counted HTTP errors as refusals, and I fixed that during validation.
• UI setup before the GPU starts. Function Calling, max_tokens, thinking off and Model IDs were finished beforehand, which the Phase 2 record says kept GPU time minimal.
• Debugging. About 4 of Phase 2's 875 seconds (about $0.12) went to the argument error. The window includes that time.
• Proving shutdown. I checked the $0/hr display after Stop, the $0 spend rate after Terminate, the pod count, and, in Phase 3, that the endpoint returned 404.
• Reconciling the bill. In Phase 3 the displayed balance dropped $3.17 against an estimate of about $2.04. The roughly $1.10 gap may be billing lag, but that is unconfirmed. The final invoice is unreconciled, and storage or volume deletion beyond what the reports cover is unverified (final retrospective).
• Everything around the model. The retrospective lists environment and dependency compatibility, authentication setup, quote and argument handling, Open WebUI settings, GPU supply, large weight transfer and storage, and shutdown and billing confirmation.

The retrospective also records "about two weeks" as my experienced calendar and preparation effort, explicitly not measured wall time, GPU hours, or benchmark time. I don't break it down further here. Of the items above, the only durations recorded are Phase 2's roughly 4 minutes on the argument error and each run's measurement window. No public record measures the rest of the preparation and checking time.

API or self-hosting: the questions I would ask

Read this in three layers:

• Public record (PR #46's retrospective): consider the MiMo-V2.6-Pro API (mimo-v2.6-pro) as an operational candidate for ordinary code, document and final-artifact review. It is unverified: no account, key, call or result, no integration, and no measured same-workload comparison with self-hosting. That status stays as recorded (final retrospective).
• My present account (not in the public record): the recurring review use case, my usage frequency and the GPU operating burden led me to choose the API as a practical direction. It is my decision, not a measured result.
• Measured results: none for the API. No calls, integration, benchmark, or cost or quality comparison, and no claim that it is cheapest, best, or ranked anywhere.

The retrospective also notes that the model's weights are published as open weights, so it isn't API-only.

The questions below are general inference, not measurements from my setup. I did not check MiMo's current pricing or the individual clauses of its provider's privacy policy against the official pages while writing this, so I leave them out.

1. How will you compare costs? Compare against an assumed usage volume. A pod bills per hour and an API per token, so the unit prices can't be compared directly. My Phase 1 to 3 figures come from connection-test windows and can't be converted into a per-token cost. Check the provider's current pricing page when you decide.
2. What happens to your data? Whether code or documents may go to an external API is a question to settle before sending anything. Check the provider's official pages for how data is retained and used, and check the applicable contract, permissions and region. I did not verify these while writing this post. An API isn't inherently private, and self-hosting isn't inherently secure.
3. How much will you run it? A pod's cost grows with the time you hold it, and an API's cost grows with the tokens you use. Which is cheaper depends on your usage, and I haven't measured it.
4. How much operating time can you spare? The previous section's non-GPU work is yours to do when you self-host. The records also show cases where I couldn't get the GPU I wanted.
5. How exposed do you want to be to change? An API's prices and behavior can change on the provider's schedule. Self-hosting means following model and serving-stack updates yourself.

Where self-hosting still earns its place

The retrospective names pinned reproducibility, control over weights, custom requirements, privacy architecture and sustained high utilization as cases where self-hosting may fit. It adds that you still need verified data handling and a measured, same-workload cost comparison. My own additions, as inference:

• You need a specific set of weights or a specific quantization. The Phase 3 trial record notes that tracks A and B share a quantization and size but use different weights, so they can be compared fairly (Phase 3 trial record). A comparison that fixes the quantization and size and varies only the weights is awkward to build on a single provider's API.
• The input can't leave your environment. Self-hosting is a strong candidate, though you still have to verify how your own environment handles the data.
• You want to adjust the serving stack itself. Server settings and launch arguments are only yours to change when you run the server.
• Usage is high and steady. If utilization is continuously high, the cost picture can shift. That still needs its own measurement.

RunPod did work for me: Phases 1 and 2 each reached one reply. Ending Phase 3 says nothing about whether RunPod is usable.

Lessons recorded in the repo

These come from the final retrospective and the phase records. They are lessons, not scheduled work.

• Check how start-command arguments split locally, and keep JSON-valued arguments out of the start command (use environment variables or a config file).
• Read the container logs in the first minute, because /health 404 or 502 can't separate a restart loop from a slow load.
• Finish all UI settings before the GPU starts.
• Fetching large weights is the main risk. The recorded ideas are to pre-fetch onto a network volume in the same region from a cheap CPU pod, to use a parallel downloader and verify SHA-256 against the registry, and to keep the GPU and the volume in the same region.
• Split the readiness stop condition into fetch and load. Mine was written for startup failures and ended up firing on download speed.
• Approve a fallback GPU and its limit up front, so a stock change doesn't stall the run.
• Define shutdown proof: a $0 spend rate after Stop, zero pods after Terminate, and the endpoint unreachable (404).
• Inject API keys from Secret references, always override a template's default key, and keep keys, endpoint URLs and pod IDs out of public docs.
• Report operational faults separately from model capability.

What this post doesn't claim

• A total cost across phases. Each figure is a window-based estimate, not a final invoice.
• A quality, speed, long-context or image-input benchmark for any model.
• Any result for the GLM models (Phase 3 produced zero model outputs).
• A cause for the slow weight download in Phase 3.
• A reason for ending the evaluation, or for moving from Colab to RunPod. The public records give neither.
• Any measurement of the MiMo-V2.6-Pro API's behavior, quality, latency or cost, any check of its data handling, or any integration status.
• MiMo's current pricing or any individual clause of its privacy policy. I didn't verify those on the official pages.
• That RunPod or any other service is unusable, or that an API is always cheaper or easier.

Sources

• Repository: moruku36/open-llm-multimodal-lab (README)
• Phase 1 verification record (Japanese only)
• Phase 2 verification record (PR #44)
• Phase 3 trial record (PR #45)
• Final retrospective (PR #46)