Last month, I saw an unexpected cost increase in the AI-powered operational pipelines of my side product. My AI API bill, which was 120 USD in the previous period, jumped to 480 USD due to a few new features I hadn't optimized. This increase was a concrete example of what we call the "Tokenpocalypse," a situation where token consumption increases uncontrollably and inflates costs. In AI projects, especially in applications running in production environments, proactively managing token costs is no longer a luxury but a necessity. In this post, I will detail seven practical methods I implemented to reduce AI bills, based on my experiences with my own applications and a client project.
How to Reduce Token Consumption with Prompt Engineering?
Every word, and even every character, we send to AI models is a cost factor. Therefore, writing prompts as short, clear, and goal-oriented as possible is the most fundamental step that directly affects token consumption. In an AI-powered production planning module I used for operator screens in a production ERP, I initially used very long and descriptive prompts. However, by optimizing the prompts later, I achieved nearly 30% token savings.
What are the Strategies for Shortening and Structuring Prompts?
When shortening prompts, it is essential not to lose the critical information the model needs. Clearly specifying the model's output format and your expectations reduces unnecessary "thinking" steps and, consequently, tokens. For example, if we expect JSON output from the model, we should explicitly state this.
💡 Prompt Tips for Token Savings
- **Be Direct:** Avoid unnecessary introductions or greetings.
- **Structured Input:** If possible, send input in structured formats like JSON or YAML.
- **Few-shot learning:** Provide short examples to help the model better understand what you want, but keep examples minimal.
- **Specify Output Format:** Clearly request the output format you expect from the model (JSON, markdown list, etc.).
Let's take an example: initially, I used a prompt like this to summarize a text:
"Hello, could you please read the text below? This text is quite long, so could you provide a short summary of 3-4 sentences for me? It's important that the summary includes the main ideas. Thank you."
This prompt contains unnecessary conversational filler and politeness. To reduce token costs, I optimized it as follows:
"Summarize the following text in 3-4 sentences. Highlight the main ideas.
Text: [Text goes here]"
Even this simple change can provide 10-15% token savings, depending on the length of the text. In the real world, these percentages create significant cost differences in large text corpuses. In an analysis I conducted on my own system, for a 1000-word text, the initial prompt consumed 150 tokens, while the optimized prompt consumed only 120 tokens. This means a difference of 300,000 tokens in a system making 10,000 calls per month.
When to Use Smaller and Specialized Models?
Using the largest and most capable AI model for every task is like driving a Ferrari to the grocery store. Often, smaller, faster, and more cost-effective models are more than sufficient for specific tasks. Especially for tasks like classification, simple text generation, or data extraction, smaller models perform very similarly to large models, and sometimes even better.
How to Balance Cost and Capability in Model Selection?
In my experience, model selection is a trade-off. General-purpose large models (GPT-4, Gemini Advanced) are expensive and slow, but excel at complex and creative tasks. Smaller models (Gemini Flash, Llama 3 8B) are cheaper and faster, but may be limited in certain capabilities. In a production ERP, when generating specific work instructions for operator screens with AI, we initially used a large model. However, I later realized that a smaller model was sufficient just to check if the instructions were in a standard format.
ℹ️ Considerations for Model Selection
- **Task Complexity:** Is it just a simple classification, or does it require creative text generation?
- **Cost:** Smaller models are generally much more affordable (e.g., Gemini Flash can be 1/5 to 1/10 the cost of larger models).
- **Latency:** Fast response times can be critical for some applications. Smaller models are generally faster.
- **Token Limit:** Smaller models usually have lower token limits, which is important for context management.
In the backend of my financial calculators, I initially used GPT-3.5 to categorize user inputs. In a scenario where I received an average of 50 requests per minute, this cost me approximately 60 USD per month. I later realized that I could perform the same task with a fine-tuned 7B version of a more affordable open-source model. When I ran this model on my own server, the cost per API call, excluding GPU costs, dropped to almost zero. In another example, in a client project, we reduced API call costs by 75% by switching from GPT-4 to Gemini Flash for simple yes/no questions. This makes a significant difference in systems with dozens of requests per second.
How to Prevent Unnecessary API Calls with Caching and Deduplication?
AI API calls are expensive, and asking the same question repeatedly means spending money unnecessarily. In my experience, one of the most effective ways to reduce your AI bill is to cache answers to previously asked questions and deduplicate similar requests. This method can be a lifesaver, especially in scenarios where the same or very similar prompts are sent repeatedly.
Approaches to Caching and Deduplicating AI Responses
Many AI applications are called repeatedly with the same or very similar inputs. For example, consider a product description summarization function. If multiple users want to summarize the same product description, instead of going to the AI API every time, we can cache the result of the first request and serve it to subsequent requests. In my Android spam application, I use AI to classify specific text patterns. If the same spam text comes repeatedly, instead of asking the AI every time, I store previously classified texts in a Redis cache.
⚠️ Risks of Caching
Caching should be used carefully, especially with dynamic and constantly changing content. Serving stale or incorrect information from the cache can negatively impact the user experience. Cache invalidation strategies should be well thought out.
For a simple caching mechanism, we can use Redis or Memcached. We can use the hash of the prompt (e.g., SHA256) as the key and store the response returned by the AI as the value.
import hashlib
import json
import redis
# pip install redis
# Redis connection
r = redis.Redis(host='localhost', port=6379, db=0)
def get_ai_response_with_cache(prompt_text, ai_model_func):
prompt_hash = hashlib.sha256(prompt_text.encode('utf-8')).hexdigest()
cached_response = r.get(prompt_hash)
if cached_response:
print("Served from cache.")
return json.loads(cached_response)
print("Sending request to AI API...")
ai_response = ai_model_func(prompt_text) # Actual AI API call
r.set(prompt_hash, json.dumps(ai_response), ex=3600) # Cache for 1 hour
return ai_response
# Example AI model function (will make actual API call)
def mock_ai_call(text):
# This part will be replaced with an actual AI API call
return {"summary": f"A summary about this text: {text[:50]}..."}
# Usage
prompt1 = "Tell me about the highest mountain in the world."
response1 = get_ai_response_with_cache(prompt1, mock_ai_call)
print(response1)
prompt2 = "Tell me about the highest mountain in the world." # Same prompt
response2 = get_ai_response_with_cache(prompt2, mock_ai_call)
print(response2)
This code snippet avoids token costs by retrieving the response from Redis on the second call for the same prompt, instead of going to the AI API. In a client project of mine, after implementing this method on April 28, we reduced the daily AI call count from 15,000 to 8,000. This meant a saving of approximately 200 USD on the monthly bill.
Smartly Managing Context with Retrieval-Augmented Generation (RAG)
One of the biggest cost drivers for large language models (LLMs) is the number of tokens kept within the context window. When asking questions about complex product trees or supply chain integration in a production ERP, directly adding all relevant documents to the prompt inflates token costs. The Retrieval-Augmented Generation (RAG) architecture is an excellent approach to solve this problem.
How RAG Works and Reduces Token Consumption
RAG essentially retrieves relevant information from a knowledge base (documents, databases, etc.) before responding to a user query and presents only this relevant information to the LLM as context. This prevents the LLM from having to "read" an entire document corpus. For example, when responding to a user's question "What are the assembly instructions for product X?", RAG only retrieves the document containing the assembly instructions for product X and sends it to the LLM, not all product manuals.
This diagram shows the basic flow of RAG. The key point is how small and targeted the context sent to the LLM is. In my own data platform, I developed an AI interface that allows users to ask questions about anonymous Turkish data. Initially, I was sending the entire data dictionary and methodologies to the LLM, which meant thousands of tokens for each query. After integrating RAG, by sending only the metadata and descriptions of relevant datasets to the LLM based on the user's query, I reduced the average token consumption by 60%. This allowed the LLM to work with a context of only 500-1000 tokens, whereas previously it required 3000-5000 tokens.
How to Optimize Flow in Multi-Agent Systems?
As AI applications become more complex, I've started using multi-agent systems where multiple AI "agents" collaborate instead of a single LLM call. For example, when performing production planning with AI in a production ERP, one agent checks raw materials, another examines production capacity, and a final agent creates the ultimate plan. In such systems, every agent consulting an LLM every time can rapidly increase costs. Flow optimization is critical to prevent these unnecessary calls.
Reducing Costs by Smartly Using Agent Patterns
One of the biggest challenges in multi-agent systems is the token consumption that occurs during agent interactions. My approach is to identify situations where each agent can solve a task "on its own" before making an LLM call, and prioritize rule-based systems or local functions in these situations. Keeping inter-agent communication to a minimum is also important.
💡 Flow Optimization Principles in Agents
- **Early Exit:** If an agent can solve a task without asking the LLM, it should exit immediately.
- **State Management:** Effectively manage state between agents to prevent redundant queries.
- **Minimal Communication:** Keep messaging between agents and intermediate results sent to the LLM to a minimum.
- **Hierarchical Structure:** Break down complex tasks into smaller, manageable sub-tasks and use the most appropriate agent/model for each sub-task.
In a client project, we set up a multi-agent system for supply chain integration. In the initial iteration, each agent (inventory control, supplier communication, logistics) made an LLM call at almost every step. This meant an average AI bill of 2,000 USD per day. By optimizing the flow, i.e., allowing agents to take direct action within certain rules (e.g., if stock is below a certain level, don't ask the supplier, just create an automatic order), we eliminated 70% of LLM calls. For example, pulling a simple stock query directly from a PostgreSQL database instead of the LLM provides immediate token savings. This means agents should have decision trees or simple if-else logic.
# Example of simple agent logic
def inventory_agent(item_id, current_stock, llm_client):
if current_stock < 100:
# Take direct action without asking the LLM
print(f"[{item_id}] Low stock ({current_stock}). Automatic order being created...")
# Call order creation function
return "Order created."
else:
# Consult LLM for a more complex decision
prompt = f"Current stock for product {item_id} is {current_stock}. What should I do to optimize the production plan?"
llm_response = llm_client.generate_text(prompt)
return llm_response
# A mock instead of a real LLM client
class MockLLMClient:
def generate_text(self, prompt):
return f"Response from LLM: '{prompt[:50]}...'"
# Usage
llm = MockLLMClient()
print(inventory_agent("PROD-001", 50, llm)) # Low stock, doesn't go to LLM
print(inventory_agent("PROD-002", 150, llm)) # Sufficient stock, goes to LLM
In this example, if the stock is low, a decision is made directly without an LLM call, thus preventing unnecessary token consumption.
Improving Efficiency in Input/Output Formatting
When working with AI models, the format of the input (prompt) we send and the output we receive directly affects token costs. Especially when working with large datasets or complex structures, formatting errors or inefficiencies can lead to unnecessary token consumption. I significantly reduced costs by making serious optimizations in this area in my own projects.
Saving Tokens by Using Efficient Input and Output Formats
In a production ERP, when processing production plans or iSCSI supply chain integration data from AI, we initially received the output in XML format. The tag structure of XML expressed the same data with many more tokens than JSON. When we switched to JSON, I saw up to a 20% reduction in output token count. Similarly, it is important to keep the data we send as input as simple as possible, avoiding unnecessary spaces, comments, or formatting characters.
ℹ️ Tips for Efficient Formatting
- **Prefer JSON:** Choose JSON over more "verbose" formats like XML.
- **Minimal Structure:** Use the least nested structures necessary.
- **Clean Unnecessary Characters:** Remove token-consuming redundancies like spaces, newlines, or comments.
- **Use Abbreviations:** If there's no loss of meaning, use abbreviations in key names (e.g., `pid` instead of `prod_id`).
Consider a scenario: We send a product's features to AI and ask it to generate a description.
Inefficient XML Input:
<product>
<id>12345</id>
<name>Smart Thermos Mug</name>
<features>
<feature>Double-Layer Insulation</feature>
<feature>Keeps Temperature for 12 Hours</feature>
<feature>500ml Capacity</feature>
</features>
<!-- This product is ideal for daily use -->
</product>
This XML input consumes quite a few tokens according to the model's tokenizer.
Optimized JSON Input:
{
"id": "12345",
"name": "Smart Thermos Mug",
"features": ["Double-Layer Insulation", "Keeps Temperature for 12 Hours", "500ml Capacity"]
}
This JSON input expresses the same information with significantly fewer tokens. In my own observation, for a 1000-word text, XML output consumed an average of 1200-1300 tokens, while the JSON converted version of the same information could remain around 900-1000 tokens. This means a saving of up to 25%. Additionally, it's important to ask the model to return only the necessary fields; for example, requesting additional information like {"summary": "...", "sentiment": "positive", "keywords": ["a", "b"]} instead of just {"summary": "..."} always leads to more token consumption. Don't ask for it unless necessary.
Multi-Provider Fallback and Smart Routing Strategies
Prices, performance, and capabilities can vary significantly among AI model providers. Relying on a single provider limits flexibility in terms of cost and creates risk in case of outages. Based on my experience with my own side products and a client project, a multi-provider strategy and smart routing are important ways to optimize costs and increase system resilience.
How to Implement Smart Routing Based on Price and Performance?
The core idea of a multi-provider strategy is to combine different AI providers (like Gemini Flash, Groq, Cerebras, OpenRouter) and route incoming requests to the most suitable provider based on the type, urgency, or cost target of the request. For example, for a simple classification task requiring a fast and cheap response, it might make sense to route to cheaper models from Groq or Cerebras, while for complex and creative text generation, it might be more logical to go to a more capable but expensive model like Gemini Advanced.
This diagram shows the basic flow of smart routing.
