Streaming, batching, and cost optimization at scale
Ship Claude at production volume. Cut spend with the Message Batches API for non-interactive workloads, prompt caching at scale, and smart model routing. Improve perceived latency with server-sent event streaming, and harden the stack with rate-limit and retry patterns.
View badge details
About This Course
Ship Claude at production volume. Server-sent event streaming for perceived latency, Message Batches API for 50% cost savings on non-interactive workloads, prompt caching at scale, rate-limit and retry patterns, and end-to-end cost optimization playbooks.By the end of this course you will be able to cut Orion Analytics's daily Claude spend by 40-60% without changing feature scope through routing, caching, batching, and streaming techniques you'll apply directly.
Course Curriculum
10 Lessons
Server-sent event streaming — what and why
By the end of this lesson you will know how Claude's server-sent event streaming actually works — the event types every stream carries (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop), how they map to a growing message object client-side, and when streaming meaningfully improves perceived latency (long generations, first-token-time-sensitive UIs) vs when it doesn't (batch pipelines). Sets up L2 hands-on where you measure the perceived-latency win.
Compare streaming vs blocking latency - Lab Exercises
By the end of this hands-on lab you will have run the same Orion prompt through Claude in blocking mode and streaming mode, measured first-token-time (FTT) and total-time in both, and computed the perceived-latency delta streaming buys — typically 5-20x FTT improvement on longer generations at the same total wall-clock — with your own numbers. Uses the PythonAI container + Claude API proxy. Anchors the streaming pattern you'll layer into CLD-AI-112 UIs.
Message Batches API — 50% off for non-realtime jobs
By the end of this lesson you will know when to reach for the Message Batches API — up to 10K requests in one batch, 50% off input + output pricing vs sync, results ready within 24 hours (typically <2 hours) — and the workloads it wins on: overnight classification jobs, nightly enrichment passes, evaluation runs. Includes the polling shape and the failure modes (per-request partial failures within a batch). Sets up L4 hands-on where you batch-classify 20 tickets.
Batch-classify 20 support tickets - Lab Exercises
By the end of this hands-on lab you will have submitted a Message Batch of 20 Orion support-ticket classifications, polled the batch to completion, retrieved per-request results, and computed the 50% cost saving vs the same 20 requests run synchronously. Uses the PythonAI container + Claude API proxy — you'll see the actual input_tokens/output_tokens reported in batch results and confirm they multiply against batch pricing, not sync pricing.
Prompt caching at scale — TTL, breakpoints, tiers
By the end of this lesson you will know prompt caching at production scale — cache_control markers on system prompts and long context blocks, the two TTL tiers (5-minute ephemeral, 1-hour extended for a small premium), where to place breakpoints so cache hits stay high, and the cost accounting (cache write is a 25% premium over base input, cache read is a 90% discount off base input). Sets up L6 hands-on where you measure cache_read hits across a multi-turn Orion conversation.
Multi-turn conversation with cached system prompt - Lab Exercises
By the end of this hands-on lab you will have run a three-turn Claude conversation with a large cached Orion system prompt — turn 1 shows cache_creation_input_tokens (paying the 25% write premium once), turns 2 and 3 show cache_read_input_tokens (paying only 10% of base for the cached prefix), and you'll compute the total cost saving over the same conversation with caching off. Uses the PythonAI container + Claude API proxy.
Cost optimization playbook
By the end of this lesson you will know the canonical order of moves to cut a Claude workload's cost — model routing (Haiku for classify, Sonnet for generate, Opus only for hard reasoning) → prompt caching (long stable prefixes) → batching (non-realtime jobs) → prompt shrinkage (remove filler and duplicated context) → tool_use for structured outputs (avoids retry loops) — with case studies showing realistic 40-80% cuts. Sets up L8 hands-on where you apply the playbook.
Optimize a hot path — before and after - Lab Exercises
By the end of this hands-on lab you will have taken a 25-call Orion ticket-classification workload from Sonnet 5 with no cache to Haiku 4.5 with a cached system prompt, measured before and after cost + accuracy, and defended the tradeoff — routing typically buys 5-8x cost savings, caching adds another 2-3x, and Haiku holds accuracy within 1-2 points of Sonnet on well-shaped classification prompts. Uses the PythonAI container + Claude API proxy.
Rate limits, retries, and backpressure
By the end of this lesson you will know how to handle rate limits gracefully — HTTP 429 with the Anthropic Retry-After header semantics, request-per-minute vs input-token-per-minute vs output-token-per-minute limits (each tracked separately per API key), exponential backoff with jitter to avoid thundering-herd retry storms, and backpressure patterns for load shedding when your queue depth grows faster than Anthropic's quota lets you drain it. Sets up L10 capstone.
Cut an Orion workload's cost by 80% capstone - Lab Exercises
By the end of this hands-on capstone you will have taken a 100-tickets-per-day Orion classification workload and cut its Claude cost by ~80% while holding accuracy — applying model routing (Haiku for classify), prompt caching (stable Orion system prompt), retry-with-jittered-backoff on 429s, and the Message Batches API for the overnight backlog. Emit a before/after cost + accuracy scorecard. Uses the PythonAI container + Claude API proxy — the CLD-AI-111 deliverable in one runnable file.
This course includes:
- 24/7 AI Instructor Support
- Live Lab Environments
- 5 Hands-on Lessons
- Completion Badge
Earn Your Badge
Complete all lessons to unlock the Streaming, Batching, and Cost Optimization at Scale achievement badge.
Streaming, Batching, and Cost Optimization at Scale
Awarded on completion of CLD-AI-111. The holder can ship Claude at production volume — SSE streaming, Message Batches API, prompt caching at scale, rate-limit handling, and end-to-end cost optimization playbooks.
Skills You'll Earn
Complete all lessons in this course to earn this badge