🐴 2026

So far in 2026, DeepEval has focused on making evaluation more reliable, observable, and easier to run across real-world LLM systems:

Tracing & observability improved with richer trace fields, better OTel exports, and deeper integration coverage
Model support expanded with new frontier and provider model entries, more accurate pricing, and safer capability handling
Component-level evals got cleaner with active-trace assertions, structured result exports, and less duplicate logging
Conversation simulation became more flexible with controller APIs, custom templates, and stronger test coverage
Docs & release tooling moved forward with the new docs site, changelog automation, and clearer tracing guides

Thank you to our contributors

First things first, DeepEval exists because of everyone who opened issues, reviewed changes, wrote docs, and merged code this year. Thank you for shaping every release with us.

August brought major improvements to the developer tool, focusing on enhanced reliability, flexibility, and user experience. The DAG metrics system was overhauled for easier construction and validation, while new end-to-end tests and stricter type checking increased robustness across the codebase. Documentation saw significant updates, including expanded TypeScript support and clearer examples, and onboarding was streamlined with a new setup wizard and dynamic questionnaires. Notably, support for voice conversation simulations and component-level evaluations was introduced, broadening the tool's capabilities for testing conversational and multimodal AI systems. Numerous bug fixes and refine

Breaking Change

v4.2.0

Updated BiasMetric, HallucinationMetric, MisuseMetric and ToxicityMetric scoring logic to align with deepeval's standard convention where higher scores indicate better performance (range: 0 to 1). Users are required to review and update their metric thresholds and evaluation code to reflect this directional change. (#3102) (Vamshi Adimalla)

New Feature

v4.1.10

Add support for voice conversation simulations, including generation and storage of conversation audio files. Users can now configure the output folder for voice simulation audio, and generated audio files are excluded from version control by default. This enables testing and evaluation of conversational AI agents with realistic voice interactions. (#3036) (Jeffrey Ip)

v4.1.9

Add a setup wizard script that allows users to easily install and configure the tool via a shell script, improving onboarding and setup experience. (#3084) (Kritin Vongthongsri)

v4.1.8

Add support for using user credentials in addition to service account keys when authenticating with Vertex AI. Users can now provide either service account or authorized user credentials via the GOOGLE_SERVICE_ACCOUNT_KEY environment variable, improving flexibility for different authentication scenarios. (#3017) (Yekta Kocaoğullar)
Add synchronous and asynchronous flushing of traces, allowing users to ensure all trace data is sent before shutdown or at specific points in their workflow. This provides more control over trace delivery and helps prevent loss of trace information in long-running or batch processes. (#3045) (Tanay)
Add support for using official MCP servers in evaluation workflows. Users can now pass MCP server instances from the official MCP package or from the internal implementation, and these will be normalized and validated automatically. This enables seamless integration with external MCP servers for tool and resource discovery in test cases. (#3046) (Tanay)

v4.1.7

Add a new TypeScript unit test architecture with Vitest integration, including example tests, configuration, and end-to-end workflows. This enables developers to write and run TypeScript tests using Vitest, with improved matcher support and trace-based evaluation. The setup includes sample test cases, golden tests, and negative test suites, making it easier to validate LLM outputs and application behavior in TypeScript projects. (#2880) (Vamshi Adimalla)
Add support for updating and deleting individual goldens in datasets via the SDK. Users can now modify or remove specific golden examples without needing to manage the entire dataset, improving flexibility and control over test data management. (#2945) (Vamshi Adimalla)
Add support for component-level evaluations in TypeScript integrations, enabling users to score individual spans and components within their applications. New examples demonstrate how to use component evals both in scripts and inside tests, allowing fine-grained assessment of LLM and tool outputs. The test matcher now supports evaluating traces produced by callback functions, making it easier to assert on component metrics directly within test cases. (#3003) (Vamshi Adimalla)

v4.1.5

Add integration with the Mastra framework for TypeScript, enabling users to run and trace Mastra agents and tools within the developer tool. Includes dedicated example and test configurations to support Mastra's ESM-first ecosystem and ensure compatibility with DeepEval tracing and observability features. (#2925) (Vamshi Adimalla)
Support server-driven login questionnaires, allowing the CLI to dynamically prompt new users with questions defined by the server during onboarding. This enables more flexible and customizable user setup flows without requiring CLI updates for new questions. (#2938) (Jeffrey Ip)
Add support for marking metrics and test cases as flaky, allowing users to flag unstable metrics or tests so their failures do not cause overall test failures. Flaky metrics are tracked separately in reports, and test case aggregation now distinguishes between regular and flaky results. Reports and pass rates now display flaky sub-counts for better visibility. Test cases marked as flaky will emit warnings instead of raising errors on failure. (#2951) (Jeffrey Ip)

Improvement

v4.1.10

Remove coercive language and PII from metric prompts to prevent provider guardrail failures. Prompts for knowledge retention, summarization, and multimodal faithfulness metrics now use neutral instructions and non-sensitive examples, improving compatibility with more LLM providers. The core evaluation logic remains unchanged, but borderline verdicts may shift slightly due to the revised prompt wording. (#3071) (Suryansh Patwa)
Removed references to retired model IDs from documentation and updated examples to use active models. This prevents users from copying examples that would result in failing requests. The changes affect the Amazon Bedrock, Anthropic, and Gemini documentation pages, ensuring only currently available models are listed and used in examples. (#3074) (Chirag Gupta)

v4.1.9

Add support for checking and reporting tool type mismatches in tool-calls metrics, providing more detailed feedback when the expected and called tool types differ. The metric now includes information about type mismatches in its reasoning and scoring, improving accuracy and transparency for users evaluating tool correctness. (#3062) (Tanay)
Fix grammar and improve clarity in the README documentation. (#3072) (MarkHe1222)
Fix typos in the Bedrock integration documentation and RAG QA tutorial to improve clarity and prevent confusion with parameter names. No functional changes were made. (#3075) (bajiu-bajiu)
Improve setup script to better handle terminal input when running via curl or on macOS, ensuring the wizard connects to the user's terminal correctly and avoids issues with stdin redirection. (#3085) (Kritin Vongthongsri)

v4.1.7

Add TypeScript documentation with language switcher, allowing users to view examples in either Python or TypeScript. Pages unsupported by the TypeScript SDK automatically switch to Python and disable TypeScript selection. All TypeScript examples are typechecked and validated for accuracy. (#2996) (Kritin Vongthongsri)

v4.1.5

Revamp DAG metrics to use a top-down construction approach, making it easier to build and validate complex conversational evaluation graphs. Adds methods for uploading and pulling metrics, improves node validation, and enhances error handling for verdict nodes. This update streamlines the creation and management of conversational DAG metrics for more robust and flexible evaluation workflows. (#2912) (Vamshi Adimalla)
Add UTM parameters to Confident AI links in llms.txt to enable better attribution and tracking of referrals from AI assistants. No content changes were made, only URL parameters were updated for analytics purposes. (#2920) (Tony Cueva Bravo)
Updated Synthesizer documentation to include the previously undocumented conversational_styling_config parameter and its fields, clarifying the constructor parameter list and providing detailed explanations for both single-turn and multi-turn styling options. Also fixed a syntax error in the StylingConfig code sample. (#2932) (jsaimanoj)
Fix typo in tool name in the step efficiency metrics documentation to improve clarity. (#2933) (huisman)
Add new end-to-end tests for the DAG API to improve reliability and coverage, and update documentation for greater clarity on node types and evaluation paths in the metrics DAG. (#2937) (Vamshi Adimalla)
Add strict mypy type checking to the telemetry package and introduce a telemetry smoke test script that exercises all event entrypoints, printing and recording payloads for inspection. This improves type safety and makes it easier to verify telemetry event emission and properties. Also, expand runtime detection tests to cover more CI and interactive scenarios. (#2990) (Jeffrey Ip)

Bug Fix

v4.1.10

Fix inconsistent delimiter handling for context and retrieval_context fields when loading CSV files, ensuring that files saved and reloaded use the same delimiter and produce correct lists. This prevents silent data corruption and ensures downstream metrics operate on properly parsed context elements. (#3049) (Koustav Manna)
Fixes logprob weighting in G-Eval by ensuring the final score token is used, preventing incorrect weighted scores when the same number appears in both reasoning and the final answer. Adds offline regression tests for repeated and single score-token outputs. No changes to public API or schema. (#3060) (Tung Lam)
Fix metrics to properly respect the include_reason flag, ensuring that reasons are not generated or LLM calls made when include_reason is set to False. Align sync and async reason generation for ConversationCompletenessMetric and TopicAdherenceMetric so that both modes behave consistently, including handling cases with no question-answer pairs without unnecessary LLM calls. This prevents unwanted API usage and ensures predictable metric outputs. (#3061) (webdev)
Fix recording of zero token counts and per-token rates on LLM spans so that legitimate zero values are preserved instead of being discarded. This ensures that free or self-hosted models with zero rates and calls with zero tokens are accurately represented in traces and downstream consumers display correct cost information. (#3069) (Tai An)

v4.1.8

Fixes an issue where nodes with multiple mutually exclusive verdict edges from the same judgement would never run, causing metric scores to be None without errors or logs. Now, such edges are counted as a single incoming edge, ensuring correct node execution and metric calculation. The fix preserves correct behavior for nodes with multiple parents or root nodes. (#3024) (Jordan Hochenbaum)
Prevent declaring a node as a root if it is also reachable from another root, ensuring DAGs do not have ambiguous entry points and avoiding duplicate execution or errors during traversal. Multiple independent roots without incoming edges remain supported. (#3027) (Jordan Hochenbaum)
Fix OpenTelemetry integrations to ensure span attributes are mutable, allowing attributes to be set or updated after span creation. This improves compatibility with tracing workflows that require dynamic attribute assignment. (#3058) (Vamshi Adimalla)

v4.1.5

Fixes silent scoring errors in MathQA, DROP, and BigBenchHard benchmarks that caused correct answers to be marked incorrect without raising errors. MathQA now correctly supports option 'e' as a valid answer, resolving systematic underreporting of accuracy for questions with this option. DROP answers containing commas are now handled properly, preventing answer corruption. BigBenchHard scoring logic is also corrected to ensure accurate evaluation. (#2916) (Muhammad Hassaan)
Fix dataset saving to correctly serialize tool call enums in JSON mode, preventing errors when saving datasets containing tool calls. Users will no longer encounter 'Object of type ToolCallType is not JSON serializable' errors, and enum values are now emitted as JSON-safe strings. (#2926) (nightcityblade)
Fix crash in EvaluationDataset.save_as when serializing ToolCall enums by ensuring enums are converted to plain strings in all supported formats. This prevents TypeError exceptions when saving datasets containing ToolCall objects. (#2928) (Solaris-star)
Fixes schema generation for gateway models by setting additionalProperties: false recursively on all nested objects, ensuring compatibility with OpenAI's strict structured-output mode. Nested pydantic models are now accepted without schema errors, preventing unnecessary fallbacks and misleading warnings. Schemas that explicitly allow extra properties remain unaffected. (#2930) (Mustafa Ensar Kulaç)
Change UTM parameter order so utm_content is stamped first, ensuring GA4's 100-character link_url captures the full label for outbound clicks. This prevents truncation of important label information in analytics and improves accuracy of click-through reporting. No changes to existing parameter handling or navigation behavior. (#2934) (Tony Cueva Bravo)
Fix system instruction parsing to handle both JSON and plain string formats, improving compatibility with different OpenTelemetry SDKs and preventing export failures due to malformed spans. (#2941) (Tanay)
Fix serialization of the tool call type to ensure correct output for any Pydantic serialization method. (#2979) (Vamshi Adimalla)

July

July's release focused on improving reliability, safety, and usability across the toolset. Several new deterministic metrics were introduced, including agent loop detection and tool permission checks, both designed for zero-latency, production-ready evaluation without external dependencies. Numerous bug fixes addressed issues with function signatures, error reporting, and hashability, while enhancements to prompt construction and verdict normalization increased consistency and clarity. Documentation and workflow improvements, including automated release note drafting and clearer guidance on model selection, further streamlined the developer experience.

New Feature

v4.0.8

Add a fully deterministic AgentLoopDetectionMetric that detects infinite loops and cyclical execution patterns in agent traces without requiring an LLM or API key. The metric analyzes tool call repetition, reasoning stagnation, and call graph cycles, returning a weighted score from 0.0 (severe looping) to 1.0 (clean execution). Designed for zero cost and zero latency, it is suitable for production use and includes comprehensive test coverage. (#2645) (Jeel3011)
Add an example demonstrating end-to-end evaluation of chatbot responses, including loading questions and responses, generating outputs, and assessing answer relevancy. This example provides a workflow for evaluating conversational AI applications and can be used as a starting point for similar projects. (#2801) (Sunil Gaurav)
Add CitationFaithfulnessMetric to evaluate whether each cited passage directly supports its associated claim, enabling detection of citation misattribution in outputs. This metric ensures that claims are attributed to the correct supporting passages, improving the accuracy of citation-based faithfulness checks. Includes documentation and tests for correct and misattributed citations. (#2811) (rsrijith)
Add deterministic ToolPermissionMetric to check if agents only call authorized tools based on allowlist and denylist policies, enabling least-privilege enforcement and CI-gateable safety checks without requiring an API key. (#2826) (Raju R)

Improvement

v4.0.8

Hardened TurnRelevancy verdict handling by enforcing verdicts as yes or no, matching other metrics. Improved normalization to handle ambiguous responses and prevent misclassification of relevance. Added an evaluation_template parameter for greater flexibility in customizing the evaluation rubric. These changes improve reliability and consistency when using native structured-output models and custom judges. (#2750) (Akshay)
Replace all direct uses of json.dumps with a serialize_to_json helper to ensure safer and more consistent JSON serialization across the codebase. This change improves handling of complex objects, nested models, and edge cases such as non-finite floats and non-string dictionary keys, reducing the risk of serialization errors. (#2781) (Vamshi Adimalla)
Fix typos in the medical chatbot tutorial pages to improve clarity and readability. (#2800) (Han S. Jung)
Add documentation warning that gpt-5 family models without log probability support cannot be used as judge models for G-Eval or ConversationalGEval, and recommend using log-probability-capable models instead. This helps prevent confusion and errors when selecting models for evaluation. (#2805) (Tran Binh Minh)
Add support for including a test_run_id when updating the current trace, enabling better tracking and association of traces with specific test runs. (#2816) (Vamshi Adimalla)
Honor the DisplayConfig.display_option setting in evaluate() to allow showing only failing or only passing test cases in the terminal output. This makes it easier to focus on relevant results, especially in CI or pre-push hooks where full output can be noisy. When all cases are filtered out, a clear note is shown. The behavior now matches the CLI's --display option. (#2829) (siddharth singh)
Fix incorrect markup in documentation to improve formatting and readability. (#2831) (Robert Lehmann)
Update release workflows to support new tag-based release process and improve changelog generation. Add a new check to enforce pull request title naming conventions, helping maintain consistency and clarity in PR titles. (#2835) (Vamshi Adimalla)
Update the marketplace configuration to reflect the new plugin name and maintain accurate ownership information. (#2852) (Vamshi Adimalla)
Add support for specifying a project ID when interacting with datasets and annotations in the TypeScript package. This enables users to associate operations with specific projects, improving organization and access control. The CLI entry point is now properly defined in the package configuration. (#2860) (Vamshi Adimalla)
Automate the drafting of release notes by integrating AI into the release workflow, streamlining the process and reducing manual effort. (#2861) (Vamshi Adimalla)
Update MCP metrics to use tools_called for improved clarity and reduce type strictness in validation, allowing for more flexible input types. This change makes it easier to integrate with different data sources and reduces errors caused by strict type requirements. (#2864) (Vamshi Adimalla)
Fix spelling errors in documentation and code comments to improve clarity and readability. No changes to functionality or APIs. (#2865) (max)
Add TypeScript support for MCP metrics to work with both toolsCalled and mcpToolsCalled, and introduce a ToolCall type enum for clearer tool call typing. This improves compatibility and type safety when handling tool calls in metrics and tracing. (#2866) (Vamshi Adimalla)

Bug Fix

v4.0.8

Fixes a function signature mismatch in the RAGAS metrics module by updating telemetry calls to include the required in_component argument, ensuring compatibility with the latest telemetry API. (#2568) (Sachin Jha)
Fixes a bug where TaskCompletionMetric would fail with a MetricTemplateInterpolationError when running with real test cases by correctly passing a formatted list of called tools to the extract_goal_and_outcome template. The tool names now reliably appear in the rendered prompt, and the no-tools case is handled cleanly. Adds tests to ensure the prompt construction includes the correct tool information and documents the previous failure mode. (#2808) (MinYi Xie)
Fixes a TypeError that occurred when hashing ToolCall instances with unhashable output or input parameters. Now, unhashable objects are safely converted to a stable string representation, allowing ToolCall objects to be used in sets and as dictionary keys without errors. Existing behavior for hashable types is preserved. (#2822) (ZentropyLabs.ai)
Prevent potential bugs by replacing the mutable list default for turn_params in convert_turn_to_dict with None and a guard, ensuring each call gets a fresh list and avoiding unintended shared state across modules. (#2827) (bono)
Fix confusing UnboundLocalError when a document yields zero chunks in context generation, now raising a clear ValueError that explains the issue. This improves error reporting for empty, whitespace-only, or unparseable files, making it easier for users to diagnose chunking problems. (#2849) (Muhammad Hassaan)
Restore support for dotted access to deepeval.evaluate.configs and its config classes while keeping deepeval.evaluate callable. This fixes AttributeError when accessing configs via deepeval.evaluate.configs and ensures compatibility with both import styles. (#2851) (ErenAta16)

June

June's release focused on expanding model support, improving documentation, and enhancing evaluation capabilities across both Python and TypeScript interfaces. Users can now work with the latest Anthropic and Gemini models, benefit from more accurate cost and token tracking, and generate synthetic contexts spanning multiple files. The documentation has been clarified and corrected throughout, with guides for custom LLMs, telemetry, and baseline test runs now more reliable and informative. Project structure and test workflows have been reorganized for better maintainability, and new features like dataset versioning and governance checks provide greater control and transparency for users.

New Feature

v4.0.7

Add support for single-turn, multi-turn, arena, and multimodal metrics in the TypeScript interface. This enables more comprehensive evaluation scenarios and allows users to work with a wider range of metric types when using the TypeScript SDK. (#2745) (Vamshi Adimalla)
Add support for Gemini 3 series models, including multimodal and structured output capabilities with updated pricing information. (#2774) (Tanay)
Add a new governance command to the TypeScript CLI, allowing users to check their project against its governance policy and receive a pass or fail verdict. The CLI provides clear output and exit codes, with options for quiet mode and help. This helps ensure projects comply with organizational governance requirements. (#2779) (Vamshi Adimalla)

v4.0.6

Add support for generating and synthesizing contexts that span multiple source files, including new configuration options to control cross-file context creation. Contexts and scenarios now track which source files they use, and prompt generation adapts to multi-file scenarios. This enables more comprehensive synthetic data generation for projects with multiple files. (#2641) (Vamshi Adimalla)
Add tracking and reporting of input and output token usage for metrics, allowing users to see token counts alongside evaluation cost. Token usage is now surfaced in metric data and tracing, providing greater transparency into resource consumption during evaluations. (#2717) (Tanay)
Add support for marking a test run as the official baseline using the --official CLI flag or the official parameter in the evaluate function. If the --official flag is used without a CONFIDENT_API_KEY environment variable, a warning is shown and the run is not marked as official. This helps users clearly designate baseline runs for Confident AI integration. (#2732) (Vamshi Adimalla)
Add support for new evaluation models in both Python and TypeScript, including improved handling of model configuration and cost calculation. The update introduces a shared gateway model base for Python, refines temperature parameter logic for Anthropic models, and adds OpenAI-compatible model support in TypeScript. These changes make it easier to use a wider range of providers and improve reliability when working with different LLM gateways. (#2738) (Vamshi Adimalla)

v4.0.5

Add support for dataset versioning, allowing users to create, retrieve, and manage multiple versions of a dataset. Users can now specify a version when pushing or pulling datasets, and list all available versions for a given dataset alias. This enables better tracking and reproducibility of dataset changes over time. (#2697) (Vamshi Adimalla)
Add support for the claude-opus-4-8 model with multimodal and structured output capabilities, as well as JSON support. (#2698) (Vamshi Adimalla)

Improvement

v4.0.7

Improve ContextualPrecisionMetric by grouping retrieved context data from the same source, reducing redundant LLM calls and providing better context to the judge model. Fix the weighted cumulative precision formula to align with documentation, ensuring more accurate scoring. (#2743) (rello-code)
Move metric template tests to a new workflow file and update documentation to reflect the new test location. This improves test organization and ensures that template synchronization checks are run independently. (#2768) (Vamshi Adimalla)
Improve serialization by using a custom Pydantic-compatible serializer as the default for all json.dumps calls, ensuring better handling of complex objects and more consistent JSON output across integrations and metrics. (#2778) (Vamshi Adimalla)

v4.0.6

Move the TypeScript package into a dedicated typescript folder, organizing related source code, configuration, and examples for improved project structure and maintainability. Adds separate linting and testing workflows for the TypeScript codebase. (#2718) (Vamshi Adimalla)
Update documentation to explain how to mark a test run as the official baseline on Confident AI using the --official flag or official=True parameter. Clarifies usage in both CLI and Python API, and notes the requirement for a Confident AI API key. (#2740) (Vamshi Adimalla)

v4.0.5

Fix broken code examples and clarify method requirements in the custom LLMs guide, ensuring users can successfully implement non-OpenAI providers without encountering errors. The guide now accurately reflects the expected method signatures and highlights common pitfalls, improving reliability for users following these instructions. (#2677) (Liz Zhang)
Corrected documentation to state that telemetry is handled by PostHog, not Sentry, and clarified the distinction between always-on telemetry and opt-in error reporting. Added a detailed list of data collected by PostHog to help users make informed opt-out decisions. (#2680) (Aadhar Agarwal)
Fix typos in tutorials and documentation to improve clarity and accuracy. No changes to tool behavior. (#2689) (Jsu)
Update documentation to include the latest Anthropic models such as claude-opus-4-8, claude-opus-4-7, claude-opus-4-5, claude-sonnet-4-6, claude-sonnet-4-5, and claude-haiku-4-5. (#2699) (Vamshi Adimalla)
Allow deepeval test run to accept any Python file, not just those starting with test_, aligning behavior with pytest and removing unnecessary filename restrictions. File and directory validation now defers to pytest, ensuring consistency and reducing friction for users with custom file naming. (#2700) (Alex Maggioni)
Fix word order in the retrieval hyperparameter note in the RAG Evaluation guide for clearer documentation. (#2709) (Tai An)
Expanded documentation to explain how users of custom models can enable GEval weighted summation, including a code example for implementing the required method. (#2713) (nuthalapativarun)
Fix several typos in the README and metrics introduction documentation to improve clarity and accuracy. No changes to code or functionality. (#2714) (Hibou04)

Bug Fix

v4.0.7

Fix cost calculation and configuration for Gemini models, allowing users to specify input and output token costs via environment variables or constructor parameters. Cost reporting now accurately reflects custom values, and documentation has been updated to describe the new options. (#2724) (Vamshi Adimalla)
Fix incorrect URL redirection in the getting started documentation so that the introduction link now points to the correct page. (#2747) (muller-j)
Fix AmazonBedrockModel to correctly store and use user-supplied per-token costs, ensuring that custom models with explicit costs return accurate values from calculate_cost. Registered models and other providers are unaffected. Adds a regression test to prevent similar issues. (#2753) (Lucas Kim)
Fix incorrect import path that caused the official tutorial to fail with a ModuleNotFoundError when running tests, ensuring the CLI works as expected. (#2755) (Andrew Tennikoff)
Fix cost and token accumulation for leaf VerdictNode to ensure accurate tracking of evaluation resources during metric execution. This prevents underreporting of costs and tokens when using DAG-based metrics. (#2756) (Tanay)
Fixes an issue where agent spans created via the pydantic-ai integration did not populate the tools_called field, which previously caused crashes in tool-aware evaluation and prevented correct metric computation. Tool calls are now correctly extracted from message history and included in agent spans, enabling proper evaluation of pydantic-ai agents. No schema or API changes were made. (#2766) (bonpiedlaroute)
Fix LangGraph integration in the TypeScript callback handler to ensure correct tracing and context management when running under a LangGraph server deployment. The handler now properly manages per-request trace creation, preventing issues when the handler is reused across multiple server requests. Documentation has been updated to clarify how to attach the callback for server deployments and the implications for trace attributes. (#2777) (Vamshi Adimalla)

v4.0.6

Fix integration tests to properly handle missing API keys, ensuring tests are skipped rather than failing when required keys are not set. Improve context restoration in LangChain integration to prevent incorrect span context after runs. Remove hardcoded test API keys from test initialization for better security and reliability. (#2719) (Vamshi Adimalla)
Fix crash in TurnFaithfulnessMetric when processing assistant turns with no factual claims, ensuring it returns a valid score and reason instead of raising a TypeError. Both synchronous and asynchronous variants now handle empty verdicts gracefully. Added unit tests to cover these cases and prevent regressions. (#2720) (Aarkin Karnik)
Fix EvaluationDataset.save_as to correctly serialize RetrievedContextData objects in retrieval_context, preventing crashes when saving datasets in json, csv, or jsonl formats. Now, all retrieval_context items are routed through the model's serializer, ensuring consistent output and compatibility with existing serialization behavior. (#2721) (Joeseph Grey)
Fix GeminiModel to correctly report evaluation costs based on actual token usage and pricing, instead of always returning zero. This ensures accurate cost reporting in the console, EvaluationCost, and dashboards, and aligns Gemini cost calculations with other providers. Tests have been updated to verify correct cost computation and prevent regressions. (#2722) (Aarkin Karnik)
Propagate all non-zero pytest exit codes to ensure test failures and errors are correctly reported in CI environments. (#2735) (Vamshi Adimalla)

v4.0.5

Fixes a bug in the async TurnContextualRecallMetric where the expected outcome was not correctly passed, ensuring retrieval context is properly evaluated. Adds a regression test using a fake LLM to catch this issue without requiring external API calls. This prevents valid conversational retrievals from being incorrectly scored as zero. (#2694) (Ritwij Aryan Parmar)
Fixes an issue where the trimAndLoadJson function could corrupt valid JSON string values containing sequences like ', ]' or ', }'. Now, valid JSON is parsed directly without modification, and the trailing comma cleanup is only applied as a fallback if parsing fails. This ensures that string values are preserved and only malformed JSON with genuine trailing commas is altered. Additional tests confirm that valid JSON round-trips unchanged and errors are raised for genuinely invalid input. (#2701) (Kyle)
Fix variable name mismatch in the metrics introduction documentation to prevent NameError when copying code examples. (#2703) (nuthalapativarun)
Fixes an issue where TurnContextualRecallMetric returned incorrect scores in async mode by ensuring the expected outcome is passed correctly. Async and sync measurement paths now behave consistently. Adds a regression test to prevent similar issues in the future. (#2708) (Parafee41)

May

May was a month of significant enhancements and new integrations for the tool, focusing on improved reliability, expanded observability, and broader compatibility. Key updates include robust error handling, support for multimodal traces, and seamless integration with platforms like Google ADK, Strands, Cursor, and Claude. The release also introduces new CLI features, prompt optimization improvements, and detailed documentation for both users and integrators. These changes collectively make the tool more powerful, flexible, and user-friendly for Python AI developers.

New Feature

v4.0.3

Add support for multimodal traces, including both images and PDFs, for LangChain and native traces. Traces now capture and expose attachments such as images and PDFs, making it easier to analyze and debug workflows involving multimodal content. Improved detection and handling of multimodal placeholders throughout the codebase. (#2672) (Vamshi Adimalla)
Add support for simulation graphs in the conversation simulator, allowing users to define flexible and customizable conversation flows using SimulationNode and related classes. The simulator now accepts a simulation_graph parameter, enabling advanced control over user turn generation and conversation structure. The previous controller parameter is now deprecated in favor of stopping_controller, with warnings issued for deprecated usage. This update improves extensibility and user control over AI. (#2678) (Jeffrey Ip)
Add support for pulling metric definitions from the server for both GEval and ConversationalGEval metrics. Users can now retrieve and instantiate metrics by name, ensuring local metric objects are synchronized with server-side definitions. This improves workflow flexibility and consistency when working with shared or remote metrics. (#2682) (Vamshi Adimalla)
Add agent skills for both Cursor and Claude, enabling integration of DeepEval evaluation, tracing, and reporting capabilities into AI applications using these platforms. Introduce a new skill for exporting raw OpenTelemetry traces from AI applications to Confident AI's Observatory without requiring the DeepEval package, supporting vendor-neutral observability. Update documentation to clarify when to use each skill and improve guidance for instrumenting AI applications. (#2684) (Vamshi Adimalla)
Add a separate skill for DeepEval native tracing, allowing users to instrument AI applications with detailed span types, tags, and metadata for improved trace visibility in Confident AI. This separation clarifies when to use DeepEval's own tracing versus OpenTelemetry or evaluation features, making it easier to choose the right instrumentation approach for Python AI apps. (#2685) (Vamshi Adimalla)

v4.0.2

Add support for using OpenRouterModel as a native evaluation provider in metric utilities. OpenRouter is now recognized in model initialization and detection, enabling seamless integration for both synchronous and asynchronous evaluation. Cost tracking and automated provider selection are also supported. (#2632) (Djalal-H)
Add google-adk and openinference-instrumentation-google-adk packages to the dependencies, enabling integration with Google ADK for supported Python versions. (#2650) (Vamshi Adimalla)
Add JSON schemas for Google ADK integration tests, enabling structured validation and improved test coverage for Google ADK-related features. (#2660) (Vamshi Adimalla)
Add integration with the Strands framework, enabling OpenTelemetry-based tracing and span collection for Strands agents. This integration supports full feature parity with other OTel-mode integrations, including synchronized trace contexts and span-level metric collection. Users can now instrument Strands agents for detailed observability and evaluation workflows. (#2661) (Vamshi Adimalla)
Add integration and documentation for Google ADK, enabling users to instrument and trace LLM agents built with Google ADK in their workflows. (#2662) (Jeffrey Ip)
Add a new inspect command to the CLI that opens a local trace view for saved test runs, allowing users to visually inspect evaluation traces in a terminal-based interface. The command supports specifying a file or folder, and provides helpful install hints if required dependencies are missing. (#2666) (Jeffrey Ip)

Improvement

v4.0.2

Improve prompt optimization by updating the COPRO algorithm to use a more efficient and streamlined approach, reducing complexity and enhancing maintainability. The new implementation simplifies configuration, improves reproducibility with better random state handling, and clarifies the optimization process for users. (#2639) (Vamshi Adimalla)
Fix minor typos in documentation to improve clarity and accuracy. (#2642) (yuri)
Add support for Apple Silicon MPS devices in BERTScorer, allowing Mac users with compatible GPUs to utilize hardware acceleration instead of defaulting to CPU. Device selection now prioritizes CUDA, then MPS, then CPU, improving performance on Apple Silicon Macs. (#2646) (Boao(Mark) Dong)
Improve documentation for installing and using skills with Claude.ai and the local CLI, and consolidate workflow instructions for the DeepEval skill into a single location. (#2649) (Vamshi Adimalla)
Add new tests for Google ADK integration covering multiple tool usage, parallel tool calls, and span-level configuration using per-call context managers. These tests ensure that metric collection and trace context features work as expected when invoking agents and tools, improving reliability and coverage of the integration. (#2654) (Jeffrey Ip)
Add provider and integration fields to tracing types, allowing more detailed tracking of model providers and integration sources in tracing data. (#2655) (Vamshi Adimalla)
Add documentation and UI components for Google ADK and improve integration tags display in the docs. The updated documentation provides clearer guidance on using AgentCore with deepeval, including step-by-step instructions, code examples, and visual enhancements for integration features such as OTel instrumentation, CI/CD evals, and traceability. (#2656) (Jeffrey Ip)
Add support for passing provider and integration fields across all integrations, ensuring that tracing and span data include this metadata. Update tests to validate the presence of these fields, improving traceability and consistency in integration outputs. (#2657) (Vamshi Adimalla)
Add regression tests to ensure the test-run OpenTelemetry tracer does not override the host application's global tracer provider, preventing unintended side effects when integrating with other telemetry setups. (#2658) (mindbomber)
Improve console output for evaluation results by providing a more detailed and visually enhanced report, including support for HTML and Markdown exports. Users can now choose to truncate passing cases in the terminal and select the output file type for saved reports. This makes it easier to review and share evaluation outcomes. (#2663) (Vamshi Adimalla)
Add documentation for the SIMBA and COPRO prompt optimization algorithms, including detailed usage instructions, configuration options, and guidance on when to use each method. Users can now find comprehensive guides for both algorithms in the documentation, making it easier to understand their features and integrate them into prompt optimization workflows. (#2664) (Vamshi Adimalla)
Update dependencies in the lock file to include new and updated packages such as linkify-it-py, mdit-py-plugins, textual, and uc-micro-py. This ensures compatibility and provides the latest features and fixes from these libraries. (#2667) (Vamshi Adimalla)
Improve the contextual precision metric to group retrieval context entries by their source when using RetrievedContextData objects. This allows for more accurate evaluation of retrieval performance when multiple contexts originate from the same source. Documentation has been updated to clarify the accepted types for retrieval_context and the grouping behavior. (#2669) (Vamshi Adimalla)
Add an automated workflow to generate and extract release notes for each new version, streamlining the release process and ensuring consistent changelog updates. (#2671) (Vamshi Adimalla)

Bug Fix

v4.0.2

Prevent crashes in trimAndLoadJson when input is None by adding a guard and raising a descriptive error, ensuring consistent error handling and clearer feedback when the evaluation LLM returns invalid output. (#2620) (bono)
Return a perfect score of 1.0 for knowledge retention when no verdicts exist, ensuring that short conversations with nothing to forget are correctly evaluated as full retention. This prevents contradictory results where the reason indicates all knowledge is retained but the score was previously 0. (#2636) (Nguyen D. Mau)
Fix FaithfulnessMetric to include ambiguous claims as contradictions when penalize_ambiguous_claims is enabled, ensuring the generated reason accurately reflects penalizations for ambiguous claims. (#2644) (miafig)
Fix incorrect import path for DeepAcyclicGraph in the DAG metric documentation to prevent ModuleNotFoundError and ensure consistency with other examples. (#2647) (Jayden Chua)
Fixes integrations to ensure trace capture failures are detected and reported, preventing silent test passes when trace data is missing. Improves CLI startup performance by lazily loading synthesizer modules only when needed. Adds an instrument function to simplify OpenTelemetry backend setup. Ensures all root spans are walked for accurate trace hydration, even on errored traces, improving dashboard diagnostics. (#2648) (Jeffrey Ip)
Fix integration tests for OpenAI agents and LangChain to ensure compatibility and correct behavior. Updates improve the handling of span types and agent extraction, leading to more reliable test outcomes and better traceability in integrations. (#2651) (Vamshi Adimalla)
Fix trace metadata handling so that user-configured trace metadata is correctly forwarded to the API trace, preventing it from being overwritten by evaluation-specific metadata. Update documentation and integration patterns to clarify the separation between trace-level and test-case metadata, and improve AgentCore integration to follow the recommended OpenTelemetry pattern for consistent metric collection and context propagation. (#2653) (Jeffrey Ip)

April

April's release focused on streamlining configuration and improving model support, with legacy options removed and new features added for both OpenAI and Anthropic models, including multimodal and structured output capabilities. The release introduced prompt branch management, enhanced tracing with better test case correlation, and expanded support for custom metadata and tags in evaluation workflows. Documentation and changelog tooling saw significant upgrades, including a new Next.js-based docs site and improved parsing for release notes. Numerous bug fixes addressed issues in tracing, schema handling, and CSV round-tripping, while new CLI commands and simulator APIs made it easier to work

Breaking Change

v3.9.9

Remove the legacy API_KEY alias and require CONFIDENT_API_KEY for Confident uploads. Update dataset loading to use metadata instead of additional_metadata, and refresh docs/examples to use SingleTurnParams for GEval evaluation parameters. (#2635) (Jeffrey Ip)

v3.9.8

Remove the observed_callback hook from assert_test and rely on the active trace when asserting against a golden. Add results_folder/results_subfolder options to control where full structured test-run JSON is saved, and deprecate per-result .log output. (#2622) (Jeffrey Ip)
Remove the metric logging manager and related configuration options, simplifying debug settings and API endpoints. Update CI to run the simulator test suite and reorganize conversation simulator tests for the new layout. (#2629) (Jeffrey Ip)

New Feature

v3.9.9

Add a generate CLI command to create synthetic goldens from documents, contexts, scratch prompts, or existing goldens, with configurable output format, concurrency, and styling options. (#2633) (Jeffrey Ip)
Add a Cursor/skills-compatible deepeval skill with templates and guidance for generating datasets, creating pytest eval suites, enabling tracing, and iterating on evaluation failures. (#2634) (Jeffrey Ip)
Add metadata and tags support to both SingleTurnParams and MultiTurnParams, making it easier to pass custom context through single-turn and conversational evaluation workflows. (#2635) (Jeffrey Ip)

v3.9.8

Add support for the claude-opus-4-7 model, including multimodal inputs, structured outputs, and JSON mode, with updated pricing metadata. (#2617) (Tanay)
Add a conversation simulator controller API with proceed()/end() decisions, plus a public SimulationTemplate. Update and expand simulator tests and CI coverage, including safer defaults when controllers return None or unexpected values. (#2628) (Jeffrey Ip)

v3.9.6

Add support for the gpt-5.4-mini model. Metrics that rely on log probabilities now detect when the model doesn’t support them and avoid failing with unexpected errors. (#2603) (Tanay)

v3.9.5

Add support for extracting confident.trace.test_case_id in ConfidentSpanExporter so OTel-exported traces can propagate testCaseId and be linked to the right test case instead of always being null. (#2570) (Alex Maggioni)
Add prompt branch support, including pushing to a specific branch and listing, creating, renaming, and deleting branches. Cache and commit lookups can now be scoped by branch to keep versions organized. (#2583) (Vamshi Adimalla)

Improvement

v3.9.9

Improve OpenAI defaults by switching the default GPT model to gpt-5.4 when no model is configured. Add model metadata for gpt-5.4 (and its snapshot alias) and update JSON output support flags for relevant models. (#2630) (Jeffrey Ip)
Improve changelog and docs parsing by supporting React-style comment markers for release note markers, PR tags, and changelog-ignore blocks, while remaining compatible with the legacy HTML comment format. (#2631) (Jeffrey Ip)

v3.9.8

Improve documentation site by migrating to a new Next.js-based setup with updated layouts and built-in search, along with refreshed docs tooling and ignores for generated build artifacts. (#2624) (Jeffrey Ip)
Improve docs site link previews and layout by adding a default Open Graph image, tightening homepage spacing, and fixing overflow/scrolling behavior in code and terminal demo blocks. (#2627) (Jeffrey Ip)

v3.9.7

Improve telemetry dependency compatibility by using PostHog 7.x on Python 3.10+ while keeping PostHog 5–6 on Python 3.9 via environment markers. (#2605) (Manoj Kumar Nagabandi)

v3.9.6

Add new tracing guides for multi-turn chatbots, RAG flows, and AI agents, including examples for grouping turns with thread_id and instrumenting spans for better end-to-end observability. (#2581) (Vamshi Adimalla)
Add a manual GitHub Actions workflow to generate changelog updates for a given year or tag range and open an update pull request automatically. (#2588) (Vamshi Adimalla)
Add optional internal tracing for metric and model methods called inside @observe spans, controlled by CONFIDENT_TRACE_INTERNAL. When enabled, key LLM generation methods and metric execution paths are captured with more detailed nested spans; when disabled, this extra instrumentation is skipped to reduce overhead. (#2589) (Kritin Vongthongsri)

v3.9.5

Add optional turn_id and test_case_id fields to tracing across supported integrations, and include them in exported trace payloads for easier correlation of multi-turn runs. (#2576) (Kritin Vongthongsri)

Bug Fix

v3.9.8

Fix multi-turn Pydantic trace input to use the most recent user message instead of the first. This prevents follow-up questions from incorrectly showing the initial question as the trace input. (#2614) (Brian Romain)
Fix a KeyError in MLLMImage.parse_multimodal_string when parsing [DEEPEVAL:IMG:<id>] markers for images that aren’t already registered. Newly created images are now kept alive for the caller so registry lookups don’t fail. (#2615) (Tanay)
Fix Anthropic Opus 4.7 requests by omitting temperature when the model does not support it, preventing API errors in both sync and async generation. (#2618) (Tanay)
Prevent evals_iterator runs from silently doing nothing by raising a clear error when no metrics are declared at any level. This avoids misleading end-of-run messages and makes missing metric configuration easier to diagnose. (#2621) (Jeffrey Ip)

v3.9.7

Fix loading single-turn golden tool calls from CSV by parsing tools_called and expected_tools as JSON objects instead of splitting by a delimiter, matching the format produced by save_as. This prevents errors and ensures tool call goldens round-trip correctly through CSV. (#2565) (Sean Kelley)
Add missing Anthropic model entries for Claude Opus 4.6 and Sonnet 4.6, including dated IDs and short aliases. Fix Opus 4.5 pricing so cost reports are no longer inflated. Restore cost tracking for default *-latest models by registering their IDs so require_costs() no longer falls back to None. (#2584) (Ajay Sai Reddy Desireddy)
Fix conversion of conversational goldens to preserve expected_outcome, preventing metrics that rely on it from failing validation or skipping evaluation after conversion. (#2598) (aerosta)
Fix OpenAI tracing spans for newer gpt-5.x/Responses API models to correctly record input/output token counts and populate per-token cost data. This prevents None values when instrumenting an OpenAI client via patch_openai_client. (#2601) (tiffanychum)
Fix PydanticAI tracing integrations by correctly classifying agent vs LLM spans and preventing mislabeling when agent attributes are present. Improve message normalization across instrumentation versions and ensure trace context is properly reset after a trace ends. (#2606) (Vamshi Adimalla)
Fix schema construction for structured outputs by correctly unwrapping Optional[...] types and detecting nested Pydantic models through full inheritance. This prevents Optional[List[int]] from being misclassified as STRING and ensures derived BaseModel types are recognized as OBJECT. (#2611) (SamSi0322)
Fix _mcp_interaction detection so MCP usage is correctly recognized under Pydantic v2. This prevents MCP-related metrics from returning near-zero scores when tools, resources, or prompts were actually called. (#2612) (SamSi0322)

v3.9.5

Fix type annotations for model and using_native_model in base metric classes by making them proper optional fields with None defaults, improving static type checking and reducing annotation-related errors. (#2574) (Tommy Beadle)
Fix docs structured data by removing the Product schema for metric pages and generating only Article schema. This avoids incorrect organization and product metadata in the rendered schema output. (#2577) (Vamshi Adimalla)

March

March was a month of significant improvements in observability, reliability, and flexibility for the tool. Major highlights include deepened OpenTelemetry and OpenInference integrations, enabling richer tracing and metric collection for agents, tools, and LLMs, with robust export and validation options. The release also focused on fixing a wide range of issues, from memory leaks and async metric isolation to improved provider selection and more accurate test result accumulation. Enhanced documentation and code formatting ensure a smoother developer experience, while new configuration options and bug fixes make the tool more adaptable and dependable across diverse environments and workflows.

New Feature

v3.9.1

Add AgentCore integration with OpenTelemetry instrumentation, including span classification and message extraction for agent, tool, and LLM traces. Support exporting telemetry via OTLP with configurable metadata, tags, and metric collection, and provide a test mode exporter for local validation. (#2534) (Vamshi Adimalla)
Add OpenInference integration to intercept OpenTelemetry spans, extract LLM/agent/tool inputs and outputs, and export traces via OTLP. Provides configurable metadata, tags, and metric collection, and surfaces clear errors when required OpenTelemetry deps or CONFIDENT_API_KEY are missing. (#2555) (Vamshi Adimalla)

v3.8.9

Add custom_column_key_values to LLMTestCase to store custom metadata as a Dict[str, str]. Accept both custom_column_key_values and customColumnKeyValues on input and serialize as customColumnKeyValues, with type validation for safer usage. (#2530) (Brian Romain)

Improvement

v3.9.3

Improve code formatting consistency by reformatting the codebase with the latest Black rules, reducing lint noise and keeping style checks stable across environments. (#2567) (Vamshi Adimalla)
Add documentation describing the updated Amazon Bedrock integration behavior, helping users configure and use Bedrock correctly after recent changes. (#2571) (Vamshi Adimalla)

v3.9.1

Add AWS AgentCore integration documentation and CI coverage, and improve span extraction and test-mode handling. Also allow tuning OTLP batch exporter settings to better control export timing and batch size. (#2544) (Vamshi Adimalla)
Fix AgentCore tracing to capture agent and trace input/output reliably, avoid duplicate traces during evaluation, and recognize additional GenAI span attributes. Also simplify instrument_agentcore by removing OTEL exporter tuning options and update docs to use evals_iterator() for end-to-end eval runs. (#2545) (Vamshi Adimalla)
Support selecting the evaluation provider via environment variables and passing a model name as a string when initializing metrics. This makes it easier to switch between OpenAI, Anthropic, Gemini, Azure, and local backends without changing code. (#2550) (Vamshi Adimalla)

v3.8.9

Support setting metric_collection on the active trace and span via the update helpers. This makes metric collection configuration consistent when updating an in-progress trace rather than only at trace creation. (#2532) (Vamshi Adimalla)

Bug Fix

v3.9.3

Fix chunk-size validation to treat collection.count() as chunk count rather than token count, preventing incorrect errors when generating contexts. Improve the guidance in the exception message with clearer suggestions to reduce chunk_size and chunk_overlap. (#2468) (Xuan-Phung Pham)
Fix initialize_model() to recognize Amazon Bedrock configuration so USE_AWS_BEDROCK_MODEL=YES no longer falls back to the default GPT model. This prevents silently using the wrong provider when Bedrock is intended. (#2537) (Parafee41)
Fix LangChain/LangGraph token usage extraction by reading usage_metadata with a fallback to legacy response_metadata, improving callback accuracy across versions. Improve test stability by retrying flaky Confident and integration tests and updating integration dependencies and fixtures. (#2557) (Vamshi Adimalla)
Fix async trace evaluation so per-trace metrics aren’t shared across concurrent tasks. This prevents concurrent runs from overwriting score, reason, and success, eliminating timing-dependent and inconsistent results. (#2559) (aerosta)
Fix pydantic-ai tracing so thread_id, name, and metadata set via the current trace context are exported on span start. Falls back to settings for compatibility and merges settings metadata with per-request metadata. (#2563) (Oluwanifemi Adeyemi)

v3.9.1

Fix Azure OpenAI keyless authentication by deferring credential checks to the OpenAI SDK. Only fail fast when an explicit credential is provided but empty, while preserving key-based auth and handling both SecretStr and string credentials consistently. (#2464) (ppon1086)
Fix a Pydantic ValidationError in KnowledgeRetentionMetric._extract_knowledges by correctly unpacking LLM response dicts when creating Knowledge objects, preventing double-wrapping and improving validation reliability. (#2513) (Diego Gómez Moreno)
Fix evaluate() in CLI runs to stop resetting the test run manager so results from multiple files are accumulated and reported together. Add a skip_reset option for manual control outside CLI mode. Ensure test case order values are always unique to prevent earlier results being overwritten or shown as skipped. (#2529) (Alex Maggioni)
Fix CrewAI tool tracing when events arrive out of order from the thread pool. Tool spans are now created and closed reliably using finished-event data, with corrected timestamps and consistent propagation of called tools to the parent span. (#2547) (Vamshi Adimalla)
Fix ConversationalGEval to make top_logprobs configurable. Add a top_logprobs parameter to the initializer (default 20) and use it in both sync and async execution paths instead of a hardcoded value. (#2549) (Szymon Cogiel)
Fix a memory leak when processing many multimodal test cases by storing _MLLM_IMAGE_REGISTRY in a weakref.WeakValueDictionary. Unreferenced MLLMImage instances are now garbage-collected automatically, preventing unbounded memory growth in large batch runs. (#2551) (eason)
Fix metric cache loading to ignore incomplete cached entries and fall back to recomputing when no score is available. Progress reporting now updates correctly when cached results are used. (#2552) (Konstantin)

v3.8.9

Fix generator tracing so observers record the final yielded item when a generator finishes without returning a value, improving captured outputs for streaming workflows. (#2514) (Vamshi Adimalla)
Fix FilterTemplate.evaluate_context examples by removing duplicate contexts and replacing them with distinct ones. Each example now has a unique input/output pairing, avoiding repeated contexts with different scores. (#2518) (Fiza Mukhtar)
Fix ContextConstructionConfig.critic_model defaulting to a new model when unset. It now falls back to the model passed to Synthesizer, so you only need to specify a custom model once when generating goldens from docs. (#2520) (Br1an)
Fix HumanEval evaluation so test assertions run against the generated function by executing the function and tests in the same sandboxed context. Also treat runtime exceptions as failures and expand allowed builtins needed by common HumanEval test cases. (#2521) (Br1an)
Fix temp ChromaDB directory cleanup on Windows by stopping the client system before calling shutil.rmtree. This releases open SQLite file handles and prevents PermissionError: [WinError 32] during teardown, with retries kept as a fallback. (#2522) (Br1an)
Fix calculate_weighted_summed_score to avoid ZeroDivisionError when all token logprobs are filtered out and the probability sum is 0. When no tokens survive filtering, it now falls back to the raw score instead of failing. (#2524) (VENKATA PRANAY BATHINI)
Fix the Goal Accuracy Score equation to remove a circular dependency. The formula now matches the implementation by averaging Goal Evaluation Score and Plan Evaluation Score as two distinct components. (#2526) (JevDev2304)

February

February's release focused on enhancing tool and prompt support, with significant improvements to schema handling and prompt versioning. The update introduced more flexible cache directory management and expanded regional support for Confident AI requests. Numerous bug fixes addressed issues in tracing, documentation, and platform compatibility, while dependency management was refined to streamline installations. Overall, these changes improve reliability, developer experience, and cross-platform robustness.

New Feature

v3.8.4

Add support for overriding the default cache directory via an environment variable, allowing you to relocate cached files without changing code. (#2455) (vection)
Add tool support to pulled and cached prompts, exposing any returned tools and converting their structured fields into a JSON Schema input_schema for easier function/tool calling. (#2466) (Vamshi Adimalla)
Add tool support to prompt push and update, including optional tools payloads. Improve tool schema handling by reusing output schema conversion and generating JSON Schema input parameters consistently, even for empty schemas. (#2474) (Vamshi Adimalla)

Improvement

v3.8.7

Fix minor typos in the Tool Correctness metric documentation, including list formatting and a missing final newline for cleaner rendering. (#2504) (nikkie)

v3.8.5

Improve prompt handling after the API migration by switching prompt identity and caching to use commit hashes and adding support for prompt commits endpoints. This helps prompts and logged hyperparameters stay consistent when versions change, and preserves tools and schema data when pulling from cache. (#2475) (Vamshi Adimalla)
Remove the opentelemetry-exporter-otlp-proto-grpc dependency from the default install to reduce required packages and keep OpenTelemetry exporter tooling out of core installs. (#2477) (Tommy Beadle)
Improve dependency compatibility by allowing Click 8.3.x (click < 8.4.0). Also adjust Linux-only pysqlite3-binary handling to prevent install failures on other platforms. (#2486) (Muhammad Faizan)
Improve prompt logging in traces by recording prompt alias, commit hash, label, and version as first-class span fields across integrations and the OTEL exporter. (#2487) (Vamshi Adimalla)
Add support for the AU data region for Confident AI requests. The CLI can now set region to AU, and API routing will automatically use AU endpoints when your API key starts with confident_au_. (#2494) (Vamshi Adimalla)

Bug Fix

v3.8.7

Fix Azure OpenAI requests to omit temperature for reasoning models that don’t support it, preventing Azure API errors. Validation now allows temperature=None, and defaults remain unchanged for standard models. (#2491) (aerosta)
Fix tests by updating the hardcoded valid trace UUID used in annotation test fixtures. (#2505) (Vamshi Adimalla)

v3.8.8

Fix the @observe decorator to capture the final return value from synchronous generator functions, not just yielded items. Also ensure the observer is closed reliably on normal completion, GeneratorExit, and errors. (#2509) (Vamshi Adimalla)

v3.8.6

Fix offline trace/span evaluation requests by sending the correct endpoints and parameters, and add overwrite_metrics support. Also allow passing chatbot_role when evaluating a thread, and avoid printing tool calls when none are present. (#2498) (Vamshi Adimalla)

v3.8.5

Fix the TaskCompletionMetric docs example to use goldens instead of golden, preventing a NameError when iterating over dataset.evals_iterator. (#2454) (Himanshu Kumar Singh)
Fix a_generate_with_schema_and_extract to handle models that return (result, cost) tuples. It now accrues cost when supported and extracts the actual result so downstream processing works without tuple parsing errors. (#2470) (Angelen)
Fix AmazonBedrockModel raising AccessDeniedException during async evaluations when AWS credentials are valid. Improves async-safe aiobotocore session and credential handling to prevent loss under concurrency while keeping sync behavior unchanged. (#2471) (Fiza Mukhtar)
Fix the conversation completeness prompt to better extract user intentions by separating multiple tasks per turn instead of summarizing them into one intention. (#2478) (Vamshi Adimalla)
Fix the knowledge retention extraction template to wrap extracted fields under a top-level data object, matching the expected JSON output format and improving parsing reliability. (#2479) (Vamshi Adimalla)
Fix OpenTelemetry span attribute setting to avoid sending None values for prompt metadata, reducing invalid or noisy telemetry attributes. (#2488) (Vamshi Adimalla)
Fix tracing prompt metadata conversion so prompt_alias, commit hash, label, and version are set consistently and don’t get overwritten by empty prompt objects. Improve prompt tests to avoid cache and alias collisions, making pull and cache behavior more reliable. (#2489) (Vamshi Adimalla)
Fix CrewAI tracing to capture prompt metadata and expected outputs on LLM spans, improve tool-span detection, and make tool completion more reliable when duplicate events or key mismatches occur. Also broaden metric lookup to support both underscored and public attribute names. (#2490) (Vamshi Adimalla)
Fix synthesizer crashes when include_expected_output=False and when max_quality_retries=0, preventing AttributeError and UnboundLocalError. Correct goldens generation so evolutions_used metadata no longer leaks across iterations. Add tests covering these scenarios. (#2493) (aerosta)
Fix a crash in deepeval view after login when telemetry doesn’t create a span. upload_and_open_link() now treats the span as optional and only sets attributes when it exists, so the command completes instead of raising an AttributeError. (#2496) (Jeremy Johnson)

Security

v3.8.5

Improve cleanup on Windows by updating safe_rmtree to use subprocess.run with argument lists instead of os.system. This handles paths with spaces more reliably and reduces the risk of command injection, making directory removal more robust across environments. (#2484) (Rin)

January

January saw major improvements in model integration, authentication, and documentation across the tool. New support for OpenRouter and Azure AD authentication expands model compatibility, while enhancements to tracing, metric collection, and error handling make integrations with LangChain, LangGraph, and Pydantic AI more robust and reliable. The release also introduces an automated changelog generator and updates documentation for clarity and easier navigation. Numerous bug fixes address edge cases in evaluation, tracing, and API routing, resulting in a smoother developer experience.

New Feature

v3.8.1

Add support for OpenRouter with an OpenAI-compatible model API and dynamic model names. Support structured outputs, configurable retries, and custom headers like HTTP-Referer and X-Title. Allow user-provided pricing with fallback to provider-reported pricing. (#2314) (Wang Junwei)
Add an automated changelog generator that builds ClickHouse-style release notes from git tags, with optional GitHub and AI enrichment. Backfill the docs changelog for 2025 to match the new year/month/category layout. (#2403) (Trevor Wilson)
Add upload() support for GEval and ConversationalGEval to send metric definitions (criteria/steps, required parameters, rubric, multi-turn) to Confident AI and store the returned metric id. (#2419) (Vamshi Adimalla)
Add a use_vertexai option to explicitly choose between Vertex AI and Gemini API-key clients when creating GeminiModel. This overrides the GOOGLE_GENAI_USE_VERTEXAI setting, including forcing False to avoid Vertex AI even if project/location are set. (#2436) (Trevor Wilson)
Add support for authenticating Azure OpenAI models using azure_ad_token or an azure_ad_token_provider, so you can use Azure AD credentials instead of an API key when desired. (#2448) (Vamshi Adimalla)

Improvement

v3.8.3

Add support for setting test_case_id in update_current_trace, and include it when serializing traces. This makes it easier to associate a trace with a specific test case in downstream processing. (#2463) (Kritin Vongthongsri)

v3.8.2

Remove the New Relic OpenTelemetry tracing exporter from telemetry. This reduces external tracing overhead and avoids requiring New Relic-related tracing setup during event capture. (#2364) (Kritin Vongthongsri)
Improve LangChain and LangGraph integrations by stabilizing tracing and metric collection. Fix schema and integration test behavior to be deterministic, representative of supported usage, and aligned with the documentation. (#2457) (Trevor Wilson)

v3.8.1

Fix grammar in the README to improve clarity when describing locally run evaluation models and metrics. (#2423) (yuri)
Add a 2024 changelog page to the documentation and link it from the changelog index and sidebar for easier navigation. Update the changelog generator default output directory to match the new docs path. (#2429) (Trevor Wilson)
Improve contextual recall verdicts by attaching the expected_output to each verdict, making results easier to interpret and debug in both sync and async runs. (#2449) (Vamshi Adimalla)
Add support for passing a confident_api_key to dataset and prompt objects, and use it automatically for push/pull/update/queue operations. This makes it easier to work with multiple API keys without relying on a single global setting. (#2453) (Vamshi Adimalla)

v3.7.9

Improve environment variable docs by clarifying how boolean flags are parsed, including accepted truthy/falsy tokens and how unset or unrecognized values fall back to defaults. Update env var tables to show 1/0/unset for boolean settings. (#2399) (Trevor Wilson)
Fix a typo in the getting started docs to improve readability and clarity. (#2404) (Neelay Shah)
Update GEval documentation to reference evaluation_steps (instead of evaluation_params) when describing which parameters should be included for accurate results. (#2406) (Vishnu Sai Teja)
Fix Gemini defaults and documentation to avoid retired model IDs. Update the default model to gemini-2.5-pro and refresh Gemini/Vertex AI docs to use current stable/preview models and *-latest aliases. Update custom LLM guide examples to gemini-2.5-flash. (#2414) (Trevor Wilson)

v3.7.8

Improve OpenTelemetry export configuration by introducing CONFIDENT_OTEL_URL (defaulting to the hosted endpoint) and using it across integrations. This makes it easier to point tracing to regional endpoints such as the EU collector via an environment variable. (#2400) (Vamshi Adimalla)

Bug Fix

v3.8.3

Fix the CrewAI integration tests by improving event loop handling in sync wrappers and correctly comparing tool usage when traces are returned as lists. This prevents failures caused by missing loops and mismatched tools-used invariants. (#2460) (Vamshi Adimalla)

v3.8.2

Fix answer relevancy scoring by raising an error when actual_output is empty or whitespace-only, preventing blank outputs from being treated as fully relevant. (#2451) (Trevor Wilson)
Fix Confident API request routing by inferring the region from the API key prefix when no region is set. EU keys (confident_eu_...) now automatically use the EU endpoint, preventing invalid API key errors. Defaults to the US endpoint when the region cannot be inferred. (#2456) (Trevor Wilson)
Fix Pydantic AI OpenTelemetry instrumentation by setting the global tracer provider when possible and warning if it’s already configured. Improve agent span detection by reading agent names from gen_ai.agent.name or pydantic_ai.agent.name and applying agent attributes consistently at span start/end. (#2459) (Vamshi Adimalla)
Fix PydanticAI tracing environment selection so it prefers the trace manager setting, then CONFIDENT_TRACE_ENVIRONMENT, and defaults to development only when neither is set. (#2462) (Vamshi Adimalla)

v3.8.1

Fix LangChain/LangGraph callback tracing to reuse active traces, restore trace/span context across async tasks, and keep correct parent-child span hierarchy. Also avoid overwriting trace metadata when values are not provided. (#2434) (Jeffrey Ip)
Fix Amazon Bedrock Converse response parsing by extracting all text content blocks and ignoring reasoningContent. Improve error messages when no text is returned, and return None for cost when pricing data is unavailable. (#2437) (Trevor Wilson)
Fix incorrect return type annotations for send_annotation() and a_send_annotation() by changing them from str to None, matching their actual no-return behavior. (#2441) (yuri)
Fix multimodal image metrics to fail fast with a clear ValueError when actual_output contains no images. Also validate expected_output when detecting multimodal test cases and improve error messaging for mismatched output image counts. (#2447) (Vamshi Adimalla)

v3.7.9

Fix batch scoring on the DROP benchmark to use quasi_contains_score, matching single prediction behavior. This prevents partial matches like '2' being incorrectly marked wrong when the gold answer includes variants such as '2, 2-yards'. (#2402) (Aadam Haq)
Fix progress updates to ignore missing tasks during teardown, preventing StopIteration when async callbacks run after a progress task is removed. Progress updates now safely become a no-op in this race condition. (#2405) (Trevor Wilson)
Fix MMLU batch_predict to require batch_generate to return a list of MultipleChoiceSchema and raise a clear TypeError otherwise, preventing inconsistent response handling. (#2408) (Aadam Haq)
Fix Gemini Vertex AI authentication to fall back to Application Default Credentials when no service account key is provided, instead of requiring GOOGLE_SERVICE_ACCOUNT_KEY. Only parse and validate the key when present to avoid unnecessary OAuth imports. (#2412) (Trevor Wilson)
Fix Synthesizer export and save for conversational goldens: to_pandas and save_as now handle both QA and conversational outputs, include the right fields, and raise an error only when neither type is present. (#2415) (Vamshi Adimalla)

On this page