Token Optimization Using Python For Prompt Development

Token optimization is critical for designing precise prompts. A Python framework can be designed to accept an English prompt as input, analyse it, remove duplicate or low-value content, and return a compact prompt that behaves like the original.

Large language models (LLMs) have made prompt development an important engineering discipline rather than a casual text-writing activity. A well-written prompt can improve accuracy, reduce ambiguity, and guide the model towards a predictable output. However, as prompts become longer, they also become more expensive, slower to process, and more difficult to maintain. This is where token optimization becomes valuable. The goal is not to make prompts artificially short, but to remove unnecessary text while preserving the original intent, context, constraints, and output expectations.

Why token optimization matters

A token is the unit of text consumed by an LLM. It may represent a word, part of a word, punctuation, whitespace, or a fragment of code. Every prompt sent to a model consumes input tokens, and every generated answer consumes output tokens. In production systems, token count directly affects cost, latency, rate limits, and the maximum amount of context that can fit into a request. OpenAI’s documentation also highlights that token counting helps estimate cost, route requests based on size, and avoid context-limit failures before an API call is made.

Token optimization is especially useful in repeated workflows such as customer support bots, retrieval-augmented generation systems, document summarizers, coding assistants, compliance reviewers, and agentic applications. In these scenarios, even a small reduction per request can produce meaningful savings when multiplied across thousands or millions of calls. More importantly, a shorter and cleaner prompt often reduces cognitive noise for the model, which can improve response consistency.

Prompt optimization techniques

Prompt optimization should be handled carefully because aggressive compression can damage meaning. Layered techniques should be used, starting with safe text cleanup and moving gradually towards semantic compression only when the user allows it. The following techniques are useful building blocks for a Python-based framework.

Remove filler phrases

Many prompts contain polite but unnecessary phrases such as “Could you please,” “I would like you to,” or “It would be great if.” These phrases are harmless in human communication, but they consume tokens without improving the model’s understanding. For example, “Could you please summarise the following article in five bullet points?” can be reduced to “Summarise the article in five bullet points.” The second version is shorter and equally clear.

Consolidate repeated instructions

A common source of wasted tokens is repeated intent. A prompt may say “be concise,” “keep the answer short,” and “avoid long explanations,” in three different places. A good optimizer should recognise that these instructions express the same constraint and retain only one clear version. The optimized instruction could simply be “Answer concisely.”

Deduplicate context

Long prompts often repeat the same background information across paragraphs. The optimizer can split the prompt into sentences or semantic chunks, compare their similarity, and remove near-duplicates. For example, “The customer is unhappy because the delivery was delayed” and “The delayed delivery made the customer dissatisfied” carry the same core meaning. The framework should keep the clearer sentence and remove the redundant one.

Preserve protected sections

Not every part of a prompt should be compressed. Legal clauses, safety rules, schema definitions, examples, API contracts, and compliance instructions may need to remain unchanged. The framework should allow users to mark protected sections so that optimization rules do not alter them. This approach is used by some open source prompt optimization tools, where protected tags help safeguard critical prompt fragments.

Convert verbose instructions into compact structure

A paragraph of instructions can sometimes be replaced by a compact list. For instance, “You should classify the ticket, identify urgency, provide a short reason, and return the result in JSON format” can become: “Return JSON with: category, urgency, reason.” Structured prompting reduces ambiguity and usually consumes fewer tokens than long-form prose.

Use semantic summarisation for long context

When a prompt includes lengthy background material, simple rule-based cleanup may not be enough. Semantic summarisation can condense the context into essential facts. This technique is useful for conversation history, meeting notes, policy excerpts, and document passages. The risk is that summarisation may omit details, so it should be used with a similarity check or human review in high-stakes workflows.

Figure 2 shows how a practical open source framework can be designed as a modular Python package.

Each module should perform one responsibility, making the framework easier to test, extend, and integrate with existing applications. The architecture may include a prompt parser, rule engine, semantic deduplicator, token counter, optimization pipeline, evaluation module, and command-line interface.

An example of a Python implementation

The following simplified implementation demonstrates the core idea. It is not a complete production framework, but it shows how a pipeline can remove filler phrases, normalise whitespace, deduplicate sentences, and report token savings.

import re
import tiktoken

class TokenCounter:
    def __init__(self, model=”gpt-4o-mini”):
        self.encoding = tiktoken.encoding_for_model(model)

    def count(self, text: str) -> int:
        return len(self.encoding.encode(text))

class PromptOptimizer:
    FILLERS = [
        r”\bcould you please\b”,
        r”\bplease make sure to\b”,
        r”\bi would like you to\b”,
        r”\bit would be great if you could\b”,
        r”\bkindly\b”
    ]
    def __init__(self, model=”gpt-4o-mini”):
        self.counter = TokenCounter(model)

    def remove_fillers(self, text: str) -> str:
        optimized = text
        for pattern in self.FILLERS:
            optimized = re.sub(pattern, “”, optimized, flags=re.IGNORECASE)
        return optimized

    def normalize_spaces(self, text: str) -> str:
        text = re.sub(r”\s+”, “ “, text)
        text = re.sub(r”\s+([.,;:])”, r”\1”, text)
        return text.strip()

    def deduplicate_sentences(self, text: str) -> str:
        sentences = re.split(r”(?<=[.!?])\s+”, text)
        seen = set()
        unique = []
        for sentence in sentences:
            key = re.sub(r”\W+”, “”, sentence.lower())
            if key and key not in seen:
                seen.add(key)
                unique.append(sentence)
        return “ “.join(unique)

    def optimize(self, prompt: str) -> dict:
        before = self.counter.count(prompt)
        text = self.remove_fillers(prompt)
        text = self.deduplicate_sentences(text)
        text = self.normalize_spaces(text)
        after = self.counter.count(text)
        return {
            “original_prompt”: prompt,
            “optimized_prompt”: text,
            “tokens_before”: before,
            “tokens_after”: after,
            “tokens_saved”: before - after,
            “saving_percent”: round(((before - after) / before) * 100, 2) if before else 0
        }

Testing the optimizer with a sample prompt

Consider the following verbose prompt:

Could you please carefully analyse the customer complaint below and make sure to identify the main issue, urgency level, and recommended next action. I would like you to provide the response in JSON format. Please make sure the response is concise and does not include unnecessary explanation.

A safe optimised version could be:

Analyze the customer complaint. Return concise JSON with: main_issue, urgency_level, recommended_next_action.

The optimized prompt removes polite phrasing, repeated conciseness instructions, and wordy transitions. It still retains the task, domain context, output format, and required fields. This is the right balance: the prompt is shorter, but the expected behaviour remains intact.

Adding semantic deduplication

Exact sentence deduplication is useful, but real prompts often repeat ideas in different words. For this reason, a mature framework should support semantic deduplication. One approach is to convert sentences into embeddings and calculate cosine similarity. If two sentences are above a chosen similarity threshold, the optimizer keeps the shorter or clearer sentence. This should be configurable because a threshold that is too low may remove important nuance.

Conceptual flow

Split prompts into sentences, generate embeddings, compare pairwise similarity, remove near-duplicates, rebuild the prompt, and validate that the required intent markers are still present. In an open source project, this component can be optional so that users may choose lightweight rule-based optimization or embedding-based optimization depending on their accuracy and dependency requirements.

Token monitoring during prompt execution

Token monitoring should be treated as a first-class feature. Before calling the model, the framework can estimate input tokens locally. After the model responds, it can capture actual usage metadata returned by the API. Together, these values help calculate savings, detect abnormal prompts, enforce budget limits, and build dashboards for product teams.

Here’s an example of request and response token monitoring:

from openai import OpenAI
import tiktoken

client = OpenAI()

def count_tokens(text, model=”gpt-4o-mini”):
encoding = tiktoken.encoding_for_model(model)
return len(encoding.encode(text))

prompt = “Summarize the incident report in five bullet points.”
estimated_request_tokens = count_tokens(prompt)

response = client.responses.create(
model=”gpt-4o-mini”,
input=prompt

)

actual_usage = response.usage
print(“Estimated request tokens:”, estimated_request_tokens)
print(“Actual input tokens:”, actual_usage.input_tokens)
print(“Actual output tokens:”, actual_usage.output_tokens)
print(“Total tokens:”, actual_usage.total_tokens)

Local tokenizers such as tiktoken are useful for fast estimation, especially for plain text. API-provided usage values are more authoritative because they include model-specific formatting and execution details. A robust framework should store both values: the estimate for preflight checks and the actual usage for billing, reporting, and continuous improvement.

Framework packaging and open source readiness

To make the framework useful to the community, the project should be packaged cleanly and documented well. A recommended structure is:

promptopt/
optimizer.py
token_counter.py
  semantic.py
validators.py
cli.py
tests/
test_optimizer.py
examples/
basic_usage.py
pyproject.toml
README.md

The command-line interface can make the tool easy to use:

promptopt optimize --input prompt.txt --model gpt-4o-mini --mode safe

The Python API can be equally simple:

from promptopt import PromptOptimizer

optimizer = PromptOptimizer(model=”gpt-4o-mini”, mode=”safe”)
result = optimizer.optimize(prompt)
print(result[“optimized_prompt”])

Validation: Ensuring the prompt still works

Token reduction is useful only if the optimized prompt preserves functionality. The framework should therefore validate each optimized prompt using three checks. First, it should confirm that mandatory intent phrases are still present. Second, it should compare the semantic similarity of the original and optimized prompt. Third, it should optionally run both prompts against a test set and compare outputs using quality metrics or human review. This creates an engineering discipline around prompt compression rather than treating it as a blind text-shortening exercise.

Here are a few best practices:

• Start with safe optimization rules before enabling aggressive semantic compression.
• Allow users to mark protected text that must never be changed.
• Measure token count before and after optimization for every prompt.
• Track actual API usage after execution, not just estimated tokens.
• Maintain test prompts for each application so that optimization does not silently degrade output quality.
• Expose configuration options such as compression level, similarity threshold, and model name.
• Keep logs of original prompt, optimized prompt, token savings, and validation status for auditability.

An open source token optimization framework for prompt development can bring discipline, repeatability, and cost awareness to LLM application engineering. By combining rule-based cleanup, duplicate intent removal, semantic deduplication, protected sections, token counting, and post-execution usage monitoring, developers can reduce prompt size without losing meaning. The most important principle is balance. A good optimizer should not simply make text shorter; it should make prompts cleaner, more economical, and easier for both humans and models to understand.