From Localhost to Production: Deploying Your Spring AI Agent

Running an AI agent on your laptop is one thing. Putting it in front of real users is another. The agent itself doesn't change — but everything around it does.

API keys need to stay out of your code. Token costs add up fast if you're not watching. A slow model call that was fine during testing becomes a timeout in production. And when something goes wrong, you need logs that actually tell you what happened.

This article covers the practical side of shipping a Spring AI agent: secrets management, cost awareness, rate limiting, timeouts, and what to monitor once you're live.

The API key problem

During development, it's tempting to put your API key directly in application.properties:

spring.ai.openai.api-key=sk-proj-abc123...

Don't do this. That file gets committed to version control. The key leaks. You get a bill.

The right pattern: read the key from an environment variable.

spring.ai.openai.api-key=${OPENAI_API_KEY}

Now start the application with the key set in the environment:

export OPENAI_API_KEY=sk-proj-abc123...
java -jar your-app.jar

On a cloud platform (Railway, Render, Fly.io, AWS), set this as a secret environment variable in the deployment settings — never in a config file that gets pushed.

What about local development? Use a .env file that is listed in .gitignore, and load it with a tool like direnv or just set the variable in your shell profile. Never commit .env.

Separating configuration by environment

Your app needs different settings in development vs production:

• Dev: verbose logging, low timeouts, a small token limit so you don't burn through a quota while testing
• Prod: tighter timeouts, higher token limits, structured logging

Spring Boot's profile system handles this cleanly.

src/main/resources/application.properties — shared defaults:

spring.ai.openai.api-key=${OPENAI_API_KEY}
app.agent.timeout-seconds=30
app.agent.max-tokens=2000

src/main/resources/application-dev.properties — development overrides:

logging.level.com.example=DEBUG
app.agent.timeout-seconds=60
app.agent.max-tokens=500

src/main/resources/application-prod.properties — production overrides:

logging.level.com.example=INFO
app.agent.timeout-seconds=20
app.agent.max-tokens=4000

Activate a profile at startup:

java -jar your-app.jar --spring.profiles.active=prod

Your agent code reads these values without knowing which profile is active:

@Value("${app.agent.timeout-seconds}")
private int timeoutSeconds;

@Value("${app.agent.max-tokens}")
private int maxTokens;

Timeouts — set them everywhere

In development, a slow model call is annoying. In production, it blocks a thread, holds a connection, and can cascade into a full outage.

Set a timeout at every layer.

On the ChatClient call:

String response = CompletableFuture
    .supplyAsync(() -> chatClient.prompt()
        .user(request.message())
        .tools(travelTools)
        .call()
        .content())
    .get(timeoutSeconds, TimeUnit.SECONDS);

On the HTTP client Spring AI uses internally, you can configure it via properties:

spring.ai.openai.chat.options.timeout=20000

On your REST endpoint with Spring MVC:

@PostMapping("/chat")
@ResponseTimeout(20000)
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
    // ...
}

Pick one consistent timeout value and apply it everywhere. Thirty seconds is a reasonable ceiling for most agent calls. If your agent needs longer, it's probably doing too much in a single request — consider breaking it into smaller steps.

Token costs — what you're actually paying for

Every call to the model costs money based on tokens — roughly three-quarters of a word each. The bill comes from two places:

• Input tokens: your system prompt + conversation history + the user's message + tool definitions
• Output tokens: the model's response

Output tokens cost more than input tokens. A verbose system prompt that runs 500 tokens fires on every single request. Multiply by 10,000 users and it becomes significant.

Where token costs hide:

How to track usage in Spring AI:

Every ChatResponse contains usage metadata:

ChatResponse chatResponse = chatClient.prompt()
    .user(request.message())
    .call()
    .chatResponse();

Usage usage = chatResponse.getMetadata().getUsage();
log.info("tokens — input: {}, output: {}, total: {}",
    usage.getPromptTokens(),
    usage.getGenerationTokens(),
    usage.getTotalTokens());

Log this on every production request. After a few days you'll see exactly where cost is coming from and whether it's growing as expected.

Rate limiting — protect yourself and the API

The model provider's API has rate limits — requests per minute and tokens per minute. Hit them and your app gets 429 errors.

Two problems to solve:

1. Protect against a single user hammering the API:

Use a simple in-memory rate limiter per user (or per IP if you don't have user identity):

@Component
public class RateLimiter {

    private final Map<String, AtomicInteger> callCount = new ConcurrentHashMap<>();
    private final Map<String, Long> windowStart = new ConcurrentHashMap<>();
    private static final int MAX_CALLS_PER_MINUTE = 10;

    public boolean allow(String userId) {
        long now = System.currentTimeMillis();
        long start = windowStart.computeIfAbsent(userId, k -> now);

        if (now - start > 60_000) {
            callCount.put(userId, new AtomicInteger(0));
            windowStart.put(userId, now);
        }

        return callCount.computeIfAbsent(userId, k -> new AtomicInteger(0))
                        .incrementAndGet() <= MAX_CALLS_PER_MINUTE;
    }
}

Check it in your controller:

if (!rateLimiter.allow(request.userId())) {
    return ResponseEntity.status(429)
        .body("Too many requests. Please wait a moment.");
}

2. Handle 429s from the provider gracefully:

try {
    return chatClient.prompt().user(message).call().content();
} catch (OpenAiHttpException e) {
    if (e.statusCode() == 429) {
        return "The service is busy right now. Please try again in a moment.";
    }
    throw e;
}

Never let a raw 429 surface to the user as a 500 error.

Retry with backoff — for transient failures

Model APIs have transient failures — brief network blips, occasional 500s. A single retry with a short wait handles most of them.

Spring Retry makes this straightforward:

<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>

Enable it in your main class:

@EnableRetry
@SpringBootApplication
public class Application { ... }

Add retry to your agent call:

@Retryable(
    retryFor = { OpenAiHttpException.class },
    maxAttempts = 3,
    backoff = @Backoff(delay = 1000, multiplier = 2)
)
public String callAgent(String message) {
    return chatClient.prompt().user(message).call().content();
}

This retries up to 3 times, waiting 1s then 2s between attempts. It only retries OpenAiHttpException — not your own application errors.

Do not retry on 429 errors with short delays — you'll just accelerate into the rate limit. Retry on 500s and network errors only.

What to log in production

Good production logs answer: "What did the agent do, for whom, and how did it go?"

Minimum log fields on every agent request:

@PostMapping("/chat")
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
    String requestId = UUID.randomUUID().toString();
    long start = System.currentTimeMillis();

    log.info("[req:{}] agent call — userId={} conversationId={}",
        requestId, request.userId(), request.conversationId());

    try {
        String response = callAgent(request.message(), request.conversationId());
        long durationMs = System.currentTimeMillis() - start;

        log.info("[req:{}] agent complete — durationMs={} responseLength={}",
            requestId, durationMs, response.length());

        return ResponseEntity.ok(response);
    } catch (TimeoutException e) {
        log.warn("[req:{}] agent timeout after {}ms", requestId, timeoutSeconds * 1000);
        return ResponseEntity.status(504).body("Request timed out.");
    } catch (Exception e) {
        log.error("[req:{}] agent error — {}", requestId, e.getMessage(), e);
        return ResponseEntity.status(500).body("Something went wrong.");
    }
}

The requestId ties every log line for a single request together. The durationMs tells you if calls are getting slower over time. The error path always logs the full stack trace.

Health checks and readiness

Before a request reaches your agent, you want to know the application is healthy. Spring Boot Actuator gives you this for free:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

management.endpoints.web.exposure.include=health,info
management.endpoint.health.show-details=when-authorized

GET /actuator/health returns {"status":"UP"} when the app is running. Your deployment platform (Render, Railway, Kubernetes) pings this endpoint — if it returns anything other than UP, traffic stops going to that instance.

Add a custom health check that verifies your model API is reachable:

@Component
public class ModelApiHealthIndicator implements HealthIndicator {

    private final ChatModel chatModel;

    public ModelApiHealthIndicator(ChatModel chatModel) {
        this.chatModel = chatModel;
    }

    @Override
    public Health health() {
        try {
            chatModel.call(new Prompt("ping"));
            return Health.up().build();
        } catch (Exception e) {
            return Health.down()
                .withDetail("reason", e.getMessage())
                .build();
        }
    }
}

If the model API is down, your health check returns DOWN and the platform can route around the broken instance or alert you.

A deployment checklist

Before you ship:

Secrets

• [ ] API key is in an environment variable, not in any committed file
• [ ] .env is in .gitignore
• [ ] No secrets appear in log output

Reliability

• [ ] Every agent call has a timeout
• [ ] 429 errors are caught and returned as readable messages
• [ ] Transient errors have retry with backoff

Cost

• [ ] Token usage is logged on every request
• [ ] System prompt is as concise as possible
• [ ] Conversation history window is bounded (chatHistoryWindowSize)

Observability

• [ ] Every request gets a unique request ID
• [ ] Duration is logged on every request
• [ ] Errors log the full stack trace

Operations

• [ ] Health check endpoint is configured
• [ ] Logs go somewhere you can actually read them (not just stdout on a server you can't SSH into)

What's next

The agent is deployed and handling real traffic. The next question is scale — what happens when many users hit the agent at the same time, when conversation histories grow large, and when you need to update the model or tools without downtime.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.