jina-embeddings-v4 as an OpenAI-Compatible Embeddings Server

jina-embeddings-v4 is a self-hosted server for the jina-embeddings-v4 embedding model with an OpenAI-compatible /v1/embeddings endpoint. It runs on a single NVIDIA GPU. An application that calls OpenAI for embeddings can call this server instead. The request and response bodies are the same, so setting the client's base URL is the only change needed.

When this is useful

• Text that cannot leave your network: the model runs on your infrastructure, and no external calls are made for embeddings after the weights are downloaded.
• Embedding a large dataset: there is no per-request charge.
• Existing OpenAI client code: the endpoint accepts the OpenAI request format and returns the OpenAI response format, so the calls in the application need no rewrite. The vectors are 2048-dimensional and unrelated to OpenAI's, so text already embedded with an OpenAI model has to be embedded again.
• Multilingual and code retrieval: one model embeds text in many languages, and source code when task is code.

What is in the image

• Model: jina-embeddings-v4, 3.8B parameters
• Output: 2048-dimensional float32 vectors
• GPU memory: 8.1 GB once the model is loaded, and over 10 GB while embedding a text of several thousand tokens
• Stack: FastAPI and sentence-transformers

Requirements

• An NVIDIA GPU with at least 10 GB of VRAM.
• An NVIDIA driver compatible with CUDA 12.4.
• Docker, Docker Compose, and the NVIDIA Container Toolkit.

Setup

Create a docker-compose.yml:

services:
  jina:
    image: edgaras0x4e/jina-embeddings-v4:latest
    ports:
      - "8081:80"
    volumes:
      - jina-cache:/root/.cache/huggingface
    environment:
      HF_HOME: /root/.cache/huggingface
      API_KEY: your-api-key-here
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  jina-cache:

docker compose up -d

The image download is about 7.8 GB, and 14.7 GB unpacked on disk. The prebuilt image needs no build step. Building from source (git clone the repo, then docker compose up -d --build) compiles flash-attn against PyTorch 2.5.1 and CUDA 12.4, which takes 10 to 20 minutes.

The weights are not in the image: on first start the server downloads about 7 GB from Hugging Face and loads them into GPU memory. On later starts the server reads the weights from the volume instead of downloading them again.

When /health returns ok, the server is ready to embed:

curl http://localhost:8081/health

{"status":"ok"}

Embedding text

curl http://localhost:8081/v1/embeddings \
  -H "Authorization: Bearer your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{"input": ["The train leaves at eight in the morning"]}'

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [-0.008472833782434464, -0.024404730647802353, 0.007389210630208254, ...],
      "index": 0
    }
  ],
  "model": "jinaai/jina-embeddings-v4",
  "usage": {"prompt_tokens": 8, "total_tokens": 8}
}

The array holds 2048 values. Three are shown.

The response uses the OpenAI list format. data holds one object per input text, each with an embedding array and its index. usage.prompt_tokens counts the input tokens. total_tokens equals it, since embedding produces no output tokens.

input also accepts a list of strings, one request for the whole batch.

The full request body:

Task adapters

The model produces a different embedding for the same text depending on the value of task. The default, text-matching, is for comparing two texts of the same kind, such as two support tickets or two product descriptions.

retrieval is for search, where a short query is matched against longer documents. Embed the documents with prompt_name set to passage and the query with prompt_name set to query:

curl http://localhost:8081/v1/embeddings \
  -H "Authorization: Bearer your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{
        "input": ["when does the train leave"],
        "task": "retrieval",
        "prompt_name": "query"
      }'

code is for source code and code search.

Using the OpenAI Python client

from openai import OpenAI

client = OpenAI(api_key="your-api-key-here", base_url="http://localhost:8081/v1")

resp = client.embeddings.create(
    model="jinaai/jina-embeddings-v4",
    input=["how long does the journey take"],
)
print(len(resp.data[0].embedding))

2048

The task and prompt_name fields are not OpenAI parameters, so the SDK passes them through extra_body:

resp = client.embeddings.create(
    model="jinaai/jina-embeddings-v4",
    input=["SELECT id, name FROM users WHERE active = 1"],
    extra_body={"task": "code"},
)

If the server runs without API_KEY, the OpenAI SDK still rejects an empty api_key string. Pass any non-empty placeholder.

Configuration

Environment variables on the jina service:

The compose file maps host port 8081 to port 80 in the container. If another service already listens on 8081, change the first number in 8081:80.

• Source: github.com/Edgaras0x4E/jina-embeddings-v4
• Docker image: hub.docker.com/r/edgaras0x4e/jina-embeddings-v4