New Features

• Accept both strings and array of strings for stop in the OpenAI-compatible inference endpoint (thanks @pragnyanramtha).
• Emit additional OpenInference attributes for Arize compatibility.

Bug Fixes

• Treat SSE body decoding errors as fatal.

New Features

• Improve error handling (e.g. status code propagation) and logging for complex streaming inferences (e.g. fallbacks).

& multiple under-the-hood and UI improvements (thanks @arisp)

Bug Fixes

• Return HTTP code 429 for rate limiting errors.
• Fixed a bug affecting ClickHouse database names with hyphens. (thanks @ianliuy!)

New Features

• Added TypeScript evaluators (for inference evaluations).
• Added support for vLLM's new reasoning field.
• Added aggregated variant usage data (tokens, cost, etc.) to the UI.
• Added inference cost data to exported OpenTelemetry traces. (thanks @kimsehwan96!)
• Added export.otlp.traces.include_content (default false) configuration field to include inference content (e.g. prompts, messages) in exported OpenTelemetry GenAI traces.

& multiple under-the-hood and UI improvements

New Features

• Add an MCP server to the gateway exposing its API in /mcp.
• Report provider prompt caching statistics via API and UI.
• Report usage statistics (e.g. tokens, latency, cost) for inference evaluations via CLI tool, API, and UI.
• Add the Prometheus metrics tensorzero_input_tokens_total and tensorzero_output_tokens_total.
• Add configuration field content_type_overrides to handle file inputs for long-tail providers.

& multiple under-the-hood and UI improvements

Bug Fixes

• Fixed a UI bug where a custom gateway base_path was not handled correctly in certain routes. (thanks @wangfenjin!)

New Features

• Started including embeddings requests in the Prometheus metrics tensorzero_requests_total and tensorzero_inferences_total.
• Added the configuration field observability.batch_writes.write_queue_capacity to enable backpressure for observability data in the gateway.

& multiple under-the-hood and UI improvements (thanks @majiayu000)!

Important

🆕 TensorZero Autopilot

TensorZero Autopilot is an automated AI engineer powered by TensorZero that analyzes LLM observability data, sets up evals, optimizes prompts and models, and runs A/B tests.

It dramatically improves the performance of LLM agents across diverse tasks:

Learn more →  Schedule a demo →

Bug Fixes

• Fixed two edge cases affecting batch inference.
• Fixed a UI bug affecting "Try with..." with inputs that include base64 files.
• Removed assistant message prefill for JSON functions + Anthropic (deprecated by Anthropic).

New Features

• Added an implementation of GEPA (automated prompt engineering) based on durable workflows.
• Allow users to specify duplicate tool calls in all_of tool evaluators to evaluate parallel tool calling.
• Allow users to specify an expiration date for API keys in the UI. (thanks @eibrahim95)
• Allow users to specify object_storage.endpoint = "env::MY_ENV_VAR" in addition to static values. (thanks @Meredith2328)

& multiple under-the-hood and UI improvements (thanks @majiayu000)!

Bug Fixes

• Fixed an UI issue that prevented certain pages from rendering when depending on historical configuration.

New Features

• Added Postgres as an alternative observability backend to ClickHouse. Postgres is the simplest way to get started; we recommend ClickHouse if you're handling >100 RPS.
• Added the openrouter::xxx short-hand for embedding models.
• Added support for per-session API keys in the browser (instead of a global environment variable) when auth is enabled.

& multiple under-the-hood and UI improvements!

Bug Fixes

• Fixed a compatibility bug with Valkey-based caching that only affected Redis.

New Features

• Added support for launching optimization workflows with dataset_name (instead of an inference query) in launch_optimization_workflow.

& multiple under-the-hood and UI improvements!