LLM Cost, Latency & Context Optimization

How Dori optimizes token spend, context windows, and model routing.

Running personal AI workflows continuously can lead to excessive costs and high latency if not carefully managed. Dori implements multi-tiered optimization techniques under the hood.

Optimization Strategies

1. Tiered Model Selection

Not every job requires a expensive reasoning model. Dori routes tasks based on node complexity:

• Fast / Lightweight Models (flash_lite / flash): Used for classification, intent detection, and data extraction.
• Frontier Reasoning Models (pro): Reserved for complex architectural synthesis, deep summary generation, and planning.

2. Context Window Discipline

To prevent context inflation:

• Raw tool logs and conversation histories are pruned and compressed before sending requests.
• Large output streams are summarized on the fly to retain core intent without ballooning context size.

3. Prompt Caching & Schema Enforcement

• Structural prompt headers remain static to leverage provider-level prompt caching.
• Tool responses enforce strict JSON schemas, preventing costly invalid-format retry cycles.