πŸ“ Join the 1st DeepEval meetup in SF, this Friday 5pm PT. RSVP on Luma β†’
πŸ’₯ Introducing JevEval: Jev-as-a-Judge for LLM evaluation. Read the post β†’

🐴 2026

So far in 2026, DeepEval has focused on making evaluation more reliable, observable, and easier to run across real-world LLM systems:

  • Tracing & observability improved with richer trace fields, better OTel exports, and deeper integration coverage
  • Model support expanded with new frontier and provider model entries, more accurate pricing, and safer capability handling
  • Component-level evals got cleaner with active-trace assertions, structured result exports, and less duplicate logging
  • Conversation simulation became more flexible with controller APIs, custom templates, and stronger test coverage
  • Docs & release tooling moved forward with the new docs site, changelog automation, and clearer tracing guides

Thank you to our contributors

First things first, DeepEval exists because of everyone who opened issues, reviewed changes, wrote docs, and merged code this year. Thank you for shaping every release with us.

+8

November

November's release focused on improving reliability, compatibility, and accuracy across the tool's evaluation, voice, and integration features. Key updates include robust handling of model parameters and prompt formats, enhanced support for new models and voice connectors, and more precise scoring and evaluation logic. Several critical bugs were fixed to prevent crashes and ensure consistent behavior across different environments and data formats. Documentation and test coverage were expanded to clarify usage and ensure correctness, while tracing and telemetry were unified under a new SDK for streamlined integration management.

New Feature

v4.3.4

  • Add support for including metadata in the turn API, allowing users to attach additional information to each turn. (#3448) (Tanay)

v4.3.3

  • Add support for the claude-sonnet-5-5 model in the model registry with correct handling of temperature and structured outputs. The model now avoids sending unsupported parameters, enables structured output configuration, and provides accurate cost calculation. Existing models remain unchanged. (#3413) (Vansh Bhardwaj)
  • Extend voice support with new connectors, including PhoneConnector, SIPConnector, LiveKitProvider, and TwilioProvider. Add support for SIP transport and media encryption options. Improve parallel call handling for connectors that do not support concurrent sessions, providing clearer error messages and guidance. Update internal APIs and tests to cover new voice connector functionality. (#3580) (Tanay)

Improvement

v4.3.4

  • Migrate all existing integrations to use the confident-trace SDK for tracing and telemetry. This change streamlines tracing setup and ensures consistent telemetry collection across integrations. Legacy Anthropic integration code has been removed in favor of confident-trace, reducing maintenance overhead and improving reliability. (#3505) (Vamshi Adimalla)
  • Update integration tests to use improved configuration and tracing logic, including changes to test utilities and schema files. Adjust test setup to prevent traces from being exported during tests and update Python version requirement in documentation to Python 3.10 or higher. Remove user ID fields from trace schemas and related tests for consistency. (#3506) (Vamshi Adimalla)

v4.3.3

  • Clarified documentation for the Tool Correctness metric to indicate it is reference-based, aligning with its requirement for expected tools. (#3466) (andreabosisio)
  • Raise the original provider's exception when all retry attempts are exhausted instead of a generic RetryError. This allows users to handle the specific error that caused the failure, improving error transparency and debugging. Tests have been updated to expect the provider's exception type. (#3542) (Vamshi Adimalla)
  • Update documentation to clarify support for voice agents and expand details on voice agent connectors and metrics. Improve descriptions of multi-turn and voice evaluation capabilities, integrations, and enterprise features. No changes to core functionality. (#3573) (Jeffrey Ip)

Bug Fix

v4.3.4

  • Fixes an issue where GEval or ConversationalGEval leaf nodes in DAGs would silently lose important configuration parameters such as rubric and strict_mode, leading to incorrect evaluation behavior. All constructor arguments are now properly preserved when copying these metric nodes, ensuring consistent scoring and evaluation logic. (#3255) (Christopher Krah)
  • Fix cache key for G-Eval metrics to include the rubric, ensuring that changes to the rubric trigger re-evaluation instead of reusing outdated cached scores. Metrics without a rubric continue to use the previous cache behavior. This prevents incorrect results when the rubric is tightened or modified. (#3579) (Jyotheeswara Reddy Gottam)
  • Fix turn-taking logic in LiveKit voice connector to ensure accurate handling of agent speech turns and transcription, preventing premature or missed turn completions during conversations. (#3590) (Tanay)
  • Fixes the Argument Correctness metric to correctly walk the trace and associate tool calls with their owning LLM spans during evaluation. This ensures that tool calls are properly attributed and evaluated even when not directly listed in the test case, improving the accuracy and reliability of argument correctness checks. (#3591) (Vamshi Adimalla)
  • Fixes an issue where the agent's turn could finish after the caller, ensuring the agent always completes its turn before the caller responds in voice simulations. This prevents overlapping turns and improves the accuracy of simulated conversations, especially in duplex voice scenarios. The update also handles edge cases where silence or call termination could previously result in incorrect turn handling. (#3593) (Tanay)
  • Fix calculation of audio start time for voice replies that contain no speech, ensuring all agent replies have a valid start time even when silent. This improves consistency in voice conversation simulations and prevents missing start times in edge cases. (#3594) (Tanay)
  • Fix WAV files generated by text-to-speech models to include correct RIFF and data sizes, ensuring accurate audio duration measurement and compatibility with transcription services. Prevents inflated transcription costs and errors caused by placeholder sizes in streamed WAV headers. Audio duration is now calculated based on actual frames, and all supported TTS models now return properly sized WAV files. (#3595) (Kritin Vongthongsri)
  • Prevent stale or removed TypeScript source files from being included in package builds by cleaning the dist directory before each build. This ensures that only current source files are shipped, resolving import errors and runtime crashes caused by outdated modules. Builds are now consistent across local and CI environments, and users no longer encounter missing module or function errors after upgrading. (#3604) (Tony)

v4.3.3

  • Stop defaulting RAGAS metric classes to the deprecated gpt-3.5-turbo model and use the library-wide default model instead. This prevents future API errors when OpenAI retires gpt-3.5-turbo. Existing code that explicitly sets the model parameter is unaffected. (#3407) (Manobhi Sriram)
  • Prevent failures on reasoning models by omitting the default temperature parameter in LiteLLMModel unless explicitly set. This change ensures compatibility with models that reject explicit temperature values and aligns behavior with other model integrations. Unit tests have been added to verify correct parameter handling. (#3431) (Udbhav Shrinet)
  • Fix crash when generating prompts by ensuring text prompts are passed as strings and multimodal prompts use the correct SDK Content objects. This prevents protobuf TypeErrors and improves compatibility with the xai_sdk chat interface. (#3437) (Christian Sidak)
  • Fix crash in add_test_cases_from_json_file when loading JSON files with null values for tools_called or expected_tools, treating null as empty lists for consistency with other loaders. This prevents TypeError and ensures round-trip compatibility when saving and loading datasets without tools. (#3470) (nancy)
  • Fixes a crash when reading structured tool results from MCP 2.x by correctly handling both the new structured_content field and the legacy structuredContent attribute. Ensures compatibility with different MCP versions and falls back to textual content when structured results are unavailable. Adds regression tests to verify correct behavior across versions. (#3533) (Anh Nguyen (Jerry))
  • Classifier-only evaluation results are now saved to the results_folder, ensuring that classification outputs are written locally even when no uploads occur. Save failures now issue warnings instead of stopping execution. Added tests confirm that classifier-only runs correctly write results to disk. (#3539) (prassu017)
  • Fixes scoring for digit-split tokens by correctly locating the full score span from the end of the response, ensuring accurate weighting of each token's logprob. Candidates that are strict prefixes of valid scores are now excluded, and candidates above the maximum score are ignored. Introduces an optional score_max parameter for precise control over valid score ranges. This results in more accurate and reliable score distributions, especially for models that emit multi-token numbers. (#3554) (mengzhe gan)
  • Detects image and PDF placeholders in conversation-level context for both ConversationalTestCase and ConversationalGolden, ensuring correct modality flags are set. This improves test-case accuracy by matching context handling with other conversational fields. The update does not affect image data mapping but ensures modality detection is consistent across all relevant fields. (#3572) (BosPrimigenious)

October

October's release focused on expanding support for audio and voice workflows, introducing new metrics for transcription accuracy and audio glitch detection, and enabling audio input across evaluation models. The handling of expectations in datasets was improved, with better import, export, and lifecycle management. Documentation saw significant enhancements in clarity, SEO, and usability, including a reorganized blog, updated guides, and a new setup wizard for streamlined onboarding. Additional improvements included more precise test run tracking, clearer error handling, and refined user journey attribution, all contributing to a more robust and user-friendly developer experience.

New Feature

v4.3.2

  • Add a setup script for installing and running the DeepEval setup wizard directly from the command line. The documentation now includes a CLI installation option alongside Python and TypeScript, with improved command labels and icons for clarity. The setup script automatically selects the latest stable release and aligns with the Python and TypeScript launchers for consistent installation behavior. (#3474) (Jeffrey Ip)

v4.3.1

  • Add a specific error for cases where a metric's prompt exceeds the evaluation model's context window, providing clearer feedback when test cases are too large to evaluate. This helps users identify and resolve issues related to prompt size limits across supported providers. (#3455) (Vamshi Adimalla)
  • Add an evaluation setup wizard accessible via deepeval setup in Python and npx deepeval setup in TypeScript. The wizard runs locally, reuses existing Confident AI keys, and is distributed as a standalone binary with integrity verification and caching. This streamlines project setup and avoids duplicate key creation. (#3472) (Jeffrey Ip)

v4.3.0

  • Add support for voice input across all evaluation models, enabling users to provide audio data for model evaluation. Models now report whether they support or require audio input, and audio token billing is handled according to provider specifications. This enhancement allows seamless integration of audio-based prompts and evaluation workflows. (#3460) (Vamshi Adimalla)
  • Add support for audio input in the ConversationalGEval metric, enabling evaluation of conversational test cases that include audio data. Users can now specify audio models and pass audio clips as part of multi-turn conversations, allowing for more comprehensive and multimodal evaluation scenarios. The metric automatically routes evaluation steps and scoring to the appropriate model based on whether audio is present. (#3462) (Vamshi Adimalla)

v4.2.9

  • Add a transcription accuracy metric to evaluate how faithfully a voice agent's speech-to-text output matches what the caller actually said, distinguishing between minor formatting differences and substantive transcription errors. (#3414) (Vamshi Adimalla)
  • Add support for reading metadata and tags fields in expectation evidence, allowing test cases to include and validate these attributes during evaluation. (#3443) (Tanay)
  • Add new metrics for evaluating transcription accuracy and detecting audio glitches such as held tones, dead streams, clipping, and abrupt ends in audio recordings. These metrics help identify and quantify issues in audio data, improving the reliability of speech and audio processing workflows. (#3453) (Yekta Kocaoğullar)

Improvement

v4.3.2

  • Improve documentation wording and organization for better clarity and readability. (#3487) (Jeffrey Ip)
  • Update all application links to use cloud.confident-ai.com instead of app.confident-ai.com in the CLI, documentation, setup wizard, examples, and tests. The CLI continues to tag links to the old host for compatibility. (#3520) (Jeffrey Ip)
  • Replace interactive frames in documentation with exported webp diagrams, simplifying the documentation and reducing dependencies. This change removes interactive board visualizations and uses static images instead, making the documentation easier to maintain and faster to load. (#3521) (Jeffrey Ip)
  • Rejects malformed API keys during login and provides clear error messages when an invalid key is entered. Cancelling the login process now fails gracefully and ensures the user is not logged in. (#3522) (Jeffrey Ip)

v4.3.1

  • Move blog posts into organized folders and migrate content images to a local media library, improving content management and reliability. Blog post URLs remain unchanged, while images are now referenced locally as WebP files for better performance. Covers now replace the image frontmatter and display as post heroes. Added an LLM evaluation guide post and improved validation scripts for content and media. (#3465) (Jeffrey Ip)
  • Polished and updated the LLM evaluation guide for improved clarity, more practical examples, and clearer explanations of key concepts. Enhanced code and data field formatting in documentation visuals for better readability. Updated author credits and improved descriptions to better reflect the guide's content and purpose. (#3467) (Jeffrey Ip)
  • Improve wording and descriptions related to Confident AI throughout documentation and the README to better clarify its features, benefits, and integration points for production AI monitoring and evaluation. (#3471) (Jeffrey Ip)
  • Add a Build with agents dropdown to the home hero section, allowing users to quickly access setup instructions and commands for integrating coding agents with their projects. The dropdown provides tabs for different agent options, copyable commands, and a link to the universal starter prompt, making it easier to get started with agent-driven workflows. (#3473) (Jeffrey Ip)

v4.3.0

  • Prevent dropout and held-tone detectors from triggering on normal speech, reducing false positives in audio glitch detection. Improved the accuracy of audio analysis by refining thresholds and adding tests to ensure pauses and natural speech patterns are not misclassified as glitches. (#3459) (Yekta Kocaoğullar)

v4.2.9

  • Improve documentation SEO by updating benchmark page titles to include 'Benchmark', refining the RAGAS metrics page title and description, and ensuring the enterprise page is included in the sitemap. Meta descriptions now use the first paragraph as a fallback, providing clearer summaries for search engines. Sidebar labels and page structure remain unchanged. (#3119) (Tony)
  • Improve tracking of user signup journeys by recording the original external referrer when a visitor arrives at the docs and carrying the full navigation trail across all Confident AI hosts. This enables better attribution of signups to their true sources and preserves the user's path through documentation and site boundaries. (#3331) (Tony)
  • Send the pulled dataset version along with test runs, allowing more precise tracking and reproducibility of test results. The dataset version is now included in test case metadata and is sent when available. If the dataset version is unknown, it is omitted from the test run payload. (#3417) (Yekta Kocaoğullar)
  • Improve expectations evaluation by updating the reasoning process and exposing verdicts as an attribute. Users can now access detailed verdicts for each expectation, and the reasoning behind evaluation outcomes is more clearly explained. This enhances transparency and debuggability when analyzing test results. (#3423) (Vamshi Adimalla)

v4.2.8

  • Add support for handling and saving expectations in datasets and goldens, allowing expectations to be loaded from and saved to files. Improves dataset import and export by including expectations in parsing and serialization. Ensures that expectations are properly managed throughout the test run lifecycle, including local and remote operations. (#3402) (Jeffrey Ip)

Bug Fix

v4.2.9

  • Fix model name resolution for LiteLLM models to correctly include the configured region and provider information. (#3424) (Tanay)

September

September's release introduces significant enhancements across API flexibility, voice simulation, and agent interoperability. Users can now update and delete golden test cases more easily, configure multiple speech providers, and record full conversation audio in simulations. The addition of agent discovery endpoints and WebRTC integration expands compatibility with AI platforms, while new storage options and improved changelog tooling streamline workflows. Privacy and trace data handling have been strengthened, and experimental support for TypeSafe AI's System One model enables advanced metric routing for early adopters.

New Feature

v4.2.6

  • Add support for including structured metadata in personas, allowing additional details such as account numbers or plans to be attached and rendered in text simulations. Improve background noise settings to accept both local file paths and http(s) URLs for audio files, enabling background audio to be loaded from remote sources as well as local files. If a background audio URL cannot be downloaded, the simulation now gracefully skips adding background noise instead of failing. (#3333) (Tanay)
  • Add support for Claude Opus 5.5 in the Anthropic model registry. The new model is registered with appropriate settings to prevent sending unsupported parameters, ensuring compatibility and correct cost calculation. Existing models and AmazonBedrockModel remain unchanged. (#3349) (Lucas Kim)

v4.2.5

  • Add ConversationalJevEval metric for evaluating multi-turn conversations using JevEval questions, enabling more comprehensive assessment of conversational test cases. (#3337) (Jeffrey Ip)

v4.2.4

  • Add tracking of docs pages visited to app signup links by including a compact site path parameter, allowing better attribution of user journeys from documentation to signup. The site path records up to 40 steps and is capped at 300 characters, ensuring efficient tracking without affecting other outbound links. (#3289) (Tony)

v4.2.3

  • Add support for a custom WebRTC connector, enabling integration with WebRTC-based voice transports. This allows users to connect to signaling endpoints via HTTP or WebSocket, exchange SDP offers and answers, and stream audio and text over WebRTC data channels. The connector handles signaling, candidate negotiation, and audio streaming, providing a flexible way to interact with WebRTC agents. (#3276) (Tanay)
  • Add support for storing test run results in a local SQLite database as an alternative to JSON files. The diagnose and inspect commands now display and accept SQLite stores, allowing users to view and manage test runs stored in a database. Environment variables and CLI options have been updated to configure and select the storage backend. Telemetry now reports the selected local storage backend. (#3281) (Jeffrey Ip)

v4.2.1

  • Add agent discovery endpoints and WebMCP tools, making the site discoverable and consumable by AI agents. Support Markdown content negotiation, provide RFC-compliant API catalogs and agent skills indexes, and expose ARD manifests with appropriate CORS headers. Introduce in-browser tools for searching docs, retrieving page markdown, and listing agent resources. Improve interoperability with AI agent platforms and enhance documentation accessibility. (#3109) (Tony)
  • Add support for configuring the maximum number of concurrent conversations in the conversation simulator, allowing users to control parallelism during simulation. Each conversation now gets its own connector instance, improving isolation and concurrency. The simulator enforces the max_concurrent limit and exposes the configured connector only when it is an instance, not a factory. (#3148) (Tanay)
  • Add support for recording full conversation audio in voice simulations, producing a stereo WAV file with caller and agent channels. Improve turn detection in duplex connectors, allowing turns to end early when a finished sentence is detected, and clarify how interruptions and barge turns are handled. Verbose logs for voice metrics now print breakdowns as indented JSON with rounded floats for easier reading. Update documentation pointers for new and changed voice simulation features. (#3163) (Tanay)

Experimental Feature

v4.2.4

  • Add experimental support for routing metric and classifier decision steps to TypeSafe AI's System One model (Jev) instead of the LLM when experimental mode is enabled. This includes QAG verdicts, DAG judgements, G-Eval scoring, and classifier label assignment, with no LLM fallback in experimental mode. Users can opt in via the experimental mode environment variable and must configure the required API key and SDK. All built-in metrics and classifiers are wired for this feature, except for a few L. (#3332) (Jeffrey Ip)

Improvement

v4.2.5

  • Update documentation for Generate Goldens From Documents to include three previously missing parameters in the Custom Golden Generation section, ensuring all configuration options are accurately described. (#3311) (Abdelrahman Mohamed Ali)

v4.2.2

  • Improve compatibility with Pydantic AI by supporting both legacy and modern agent instrumentation patterns, ensuring correct extraction of tool arguments and results for all supported versions. Update documentation and examples to reflect changes in Pydantic AI's API and clarify integration usage. Enhance test coverage for tool telemetry extraction and agent setup. (#3252) (Jeffrey Ip)

v4.2.1

  • Add support for updating and deleting golden test cases via new API endpoints, allowing users to modify or remove goldens from datasets more easily. The update and delete methods now require both the golden ID and dataset alias, improving clarity and error handling when managing goldens. (#3020) (Vamshi Adimalla)
  • Update the changelog generation script to use the gpt-4.1 model and improve compatibility with monorepo tag schemes. The script now correctly handles SDK-specific tag prefixes and ensures changelog entries display consistent version formatting. Existing PRs are tracked more reliably to prevent duplicate entries across releases. Command-line arguments now support specifying multiple tags for release generation. (#3105) (Vamshi Adimalla)
  • Update the documentation banner to inform users that some metrics' scoring logic now follows the convention that higher scores are better, with a link to the changelog for details. (#3107) (Vamshi Adimalla)
  • Add environment variables and configuration options for multiple speech-to-text and text-to-speech providers, including ElevenLabs, Deepgram, Cartesia, and AssemblyAI, allowing users to select and configure their preferred voice provider. Update documentation pointers for new voice mode features and connector behaviors. Improve CLI structure by introducing a root Typer app for easier command registration. Update import paths for speech model modules to reflect new organization. Add .gitignore, . (#3139) (Jeffrey Ip)

Bug Fix

v4.2.7

  • Fixes issues with phone call lifecycle in voice simulations, ensuring that calls end correctly when the agent disconnects or hangs up. Simulations now accurately record the stop reason when the agent ends the call, and agent speech that occurs before the caller replies is properly handled as its own turn. These changes improve the reliability and accuracy of voice-based conversation simulations. (#3379) (Tanay)

v4.2.4

  • Prevent sensitive instance data from being captured in async functions decorated with observe by replacing self with the class name in tracing inputs. This improves privacy and ensures that secrets such as API keys are not inadvertently logged during tracing. (#3300) (Yekta Kocaoğullar)

v4.2.2

  • Fixes the application of tools_called and expected_tools fields in OpenTelemetry integrations, ensuring these attributes are correctly set on spans when present. Improves trace data completeness and consistency for tool usage tracking. (#3103) (Vamshi Adimalla)

August

August brought significant improvements to the developer tool, focusing on enhanced evaluation workflows, expanded TypeScript support, and better onboarding experiences. Key metrics were updated to align with standard scoring conventions, and new features such as voice conversation simulations and component-level evaluations were introduced. The release also addressed numerous usability and reliability issues, including improved authentication options, more robust trace handling, and clearer documentation. These updates collectively provide users with greater flexibility, accuracy, and control in building and evaluating AI applications.

Breaking Change

v4.2.0

New Feature

v4.1.10

  • Add support for voice conversation simulations, including generation and storage of conversation audio files. Users can now configure the output folder for voice simulation audio, and generated audio files are excluded from version control by default. This enables testing and evaluation of conversational AI agents with realistic voice interactions. (#3036) (Jeffrey Ip)

v4.1.9

  • Add a setup wizard script that allows users to easily install and configure the tool via a shell script, improving onboarding and setup experience. (#3084) (Kritin Vongthongsri)

v4.1.8

  • Add support for using user credentials in addition to service account keys when authenticating with Vertex AI. Users can now provide either service account or authorized user credentials via the GOOGLE_SERVICE_ACCOUNT_KEY environment variable, improving flexibility for different authentication scenarios. (#3017) (Yekta Kocaoğullar)
  • Add synchronous and asynchronous flushing of traces, allowing users to ensure all trace data is sent before shutdown or at specific points in their workflow. This provides more control over trace delivery and helps prevent loss of trace information in long-running or batch processes. (#3045) (Tanay)
  • Add support for using official MCP servers in evaluation workflows. Users can now pass MCP server instances from the official MCP package or from the internal implementation, and these will be normalized and validated automatically. This enables seamless integration with external MCP servers for tool and resource discovery in test cases. (#3046) (Tanay)

v4.1.7

  • Add a new TypeScript unit test architecture with Vitest integration, including example tests, configuration, and end-to-end workflows. This enables developers to write and run TypeScript tests using Vitest, with improved matcher support and trace-based evaluation. The setup includes sample test cases, golden tests, and negative test suites, making it easier to validate LLM outputs and application behavior in TypeScript projects. (#2880) (Vamshi Adimalla)
  • Add support for updating and deleting individual goldens in datasets via the SDK. Users can now modify or remove specific golden examples without needing to manage the entire dataset, improving flexibility and control over test data management. (#2945) (Vamshi Adimalla)
  • Add support for component-level evaluations in TypeScript integrations, enabling users to score individual spans and components within their applications. New examples demonstrate how to use component evals both in scripts and inside tests, allowing fine-grained assessment of LLM and tool outputs. The test matcher now supports evaluating traces produced by callback functions, making it easier to assert on component metrics directly within test cases. (#3003) (Vamshi Adimalla)

v4.1.5

  • Add integration with the Mastra framework for TypeScript, enabling users to run and trace Mastra agents and tools within the developer tool. Includes dedicated example and test configurations to support Mastra's ESM-first ecosystem and ensure compatibility with DeepEval tracing and observability features. (#2925) (Vamshi Adimalla)
  • Support server-driven login questionnaires, allowing the CLI to dynamically prompt new users with questions defined by the server during onboarding. This enables more flexible and customizable user setup flows without requiring CLI updates for new questions. (#2938) (Jeffrey Ip)
  • Add support for marking metrics and test cases as flaky, allowing users to flag unstable metrics or tests so their failures do not cause overall test failures. Flaky metrics are tracked separately in reports, and test case aggregation now distinguishes between regular and flaky results. Reports and pass rates now display flaky sub-counts for better visibility. Test cases marked as flaky will emit warnings instead of raising errors on failure. (#2951) (Jeffrey Ip)

Improvement

v4.1.10

  • Remove coercive language and PII from metric prompts to prevent provider guardrail failures. Prompts for knowledge retention, summarization, and multimodal faithfulness metrics now use neutral instructions and non-sensitive examples, improving compatibility with more LLM providers. The core evaluation logic remains unchanged, but borderline verdicts may shift slightly due to the revised prompt wording. (#3071) (Suryansh Patwa)
  • Removed references to retired model IDs from documentation and updated examples to use active models. This prevents users from copying examples that would result in failing requests. The changes affect the Amazon Bedrock, Anthropic, and Gemini documentation pages, ensuring only currently available models are listed and used in examples. (#3074) (Chirag Gupta)

v4.1.9

  • Add support for checking and reporting tool type mismatches in tool-calls metrics, providing more detailed feedback when the expected and called tool types differ. The metric now includes information about type mismatches in its reasoning and scoring, improving accuracy and transparency for users evaluating tool correctness. (#3062) (Tanay)
  • Fix grammar and improve clarity in the README documentation. (#3072) (MarkHe1222)
  • Fix typos in the Bedrock integration documentation and RAG QA tutorial to improve clarity and prevent confusion with parameter names. No functional changes were made. (#3075) (bajiu-bajiu)
  • Improve setup script to better handle terminal input when running via curl or on macOS, ensuring the wizard connects to the user's terminal correctly and avoids issues with stdin redirection. (#3085) (Kritin Vongthongsri)

v4.1.7

  • Add TypeScript documentation with language switcher, allowing users to view examples in either Python or TypeScript. Pages unsupported by the TypeScript SDK automatically switch to Python and disable TypeScript selection. All TypeScript examples are typechecked and validated for accuracy. (#2996) (Kritin Vongthongsri)

v4.1.5

  • Revamp DAG metrics to use a top-down construction approach, making it easier to build and validate complex conversational evaluation graphs. Adds methods for uploading and pulling metrics, improves node validation, and enhances error handling for verdict nodes. This update streamlines the creation and management of conversational DAG metrics for more robust and flexible evaluation workflows. (#2912) (Vamshi Adimalla)
  • Add UTM parameters to Confident AI links in llms.txt to enable better attribution and tracking of referrals from AI assistants. No content changes were made, only URL parameters were updated for analytics purposes. (#2920) (Tony Cueva Bravo)
  • Updated Synthesizer documentation to include the previously undocumented conversational_styling_config parameter and its fields, clarifying the constructor parameter list and providing detailed explanations for both single-turn and multi-turn styling options. Also fixed a syntax error in the StylingConfig code sample. (#2932) (jsaimanoj)
  • Fix typo in tool name in the step efficiency metrics documentation to improve clarity. (#2933) (huisman)
  • Add new end-to-end tests for the DAG API to improve reliability and coverage, and update documentation for greater clarity on node types and evaluation paths in the metrics DAG. (#2937) (Vamshi Adimalla)
  • Add strict mypy type checking to the telemetry package and introduce a telemetry smoke test script that exercises all event entrypoints, printing and recording payloads for inspection. This improves type safety and makes it easier to verify telemetry event emission and properties. Also, expand runtime detection tests to cover more CI and interactive scenarios. (#2990) (Jeffrey Ip)

Bug Fix

v4.1.10

  • Fix inconsistent delimiter handling for context and retrieval_context fields when loading CSV files, ensuring that files saved and reloaded use the same delimiter and produce correct lists. This prevents silent data corruption and ensures downstream metrics operate on properly parsed context elements. (#3049) (Koustav Manna)
  • Fixes logprob weighting in G-Eval by ensuring the final score token is used, preventing incorrect weighted scores when the same number appears in both reasoning and the final answer. Adds offline regression tests for repeated and single score-token outputs. No changes to public API or schema. (#3060) (Tung Lam)
  • Fix metrics to properly respect the include_reason flag, ensuring that reasons are not generated or LLM calls made when include_reason is set to False. Align sync and async reason generation for ConversationCompletenessMetric and TopicAdherenceMetric so that both modes behave consistently, including handling cases with no question-answer pairs without unnecessary LLM calls. This prevents unwanted API usage and ensures predictable metric outputs. (#3061) (webdev)
  • Fix recording of zero token counts and per-token rates on LLM spans so that legitimate zero values are preserved instead of being discarded. This ensures that free or self-hosted models with zero rates and calls with zero tokens are accurately represented in traces and downstream consumers display correct cost information. (#3069) (Tai An)

v4.1.8

  • Fixes an issue where nodes with multiple mutually exclusive verdict edges from the same judgement would never run, causing metric scores to be None without errors or logs. Now, such edges are counted as a single incoming edge, ensuring correct node execution and metric calculation. The fix preserves correct behavior for nodes with multiple parents or root nodes. (#3024) (Jordan Hochenbaum)
  • Prevent declaring a node as a root if it is also reachable from another root, ensuring DAGs do not have ambiguous entry points and avoiding duplicate execution or errors during traversal. Multiple independent roots without incoming edges remain supported. (#3027) (Jordan Hochenbaum)
  • Fix OpenTelemetry integrations to ensure span attributes are mutable, allowing attributes to be set or updated after span creation. This improves compatibility with tracing workflows that require dynamic attribute assignment. (#3058) (Vamshi Adimalla)

v4.1.5

  • Fixes silent scoring errors in MathQA, DROP, and BigBenchHard benchmarks that caused correct answers to be marked incorrect without raising errors. MathQA now correctly supports option 'e' as a valid answer, resolving systematic underreporting of accuracy for questions with this option. DROP answers containing commas are now handled properly, preventing answer corruption. BigBenchHard scoring logic is also corrected to ensure accurate evaluation. (#2916) (Muhammad Hassaan)
  • Fix dataset saving to correctly serialize tool call enums in JSON mode, preventing errors when saving datasets containing tool calls. Users will no longer encounter 'Object of type ToolCallType is not JSON serializable' errors, and enum values are now emitted as JSON-safe strings. (#2926) (nightcityblade)
  • Fix crash in EvaluationDataset.save_as when serializing ToolCall enums by ensuring enums are converted to plain strings in all supported formats. This prevents TypeError exceptions when saving datasets containing ToolCall objects. (#2928) (Solaris-star)
  • Fixes schema generation for gateway models by setting additionalProperties: false recursively on all nested objects, ensuring compatibility with OpenAI's strict structured-output mode. Nested pydantic models are now accepted without schema errors, preventing unnecessary fallbacks and misleading warnings. Schemas that explicitly allow extra properties remain unaffected. (#2930) (Mustafa Ensar KulaΓ§)
  • Change UTM parameter order so utm_content is stamped first, ensuring GA4's 100-character link_url captures the full label for outbound clicks. This prevents truncation of important label information in analytics and improves accuracy of click-through reporting. No changes to existing parameter handling or navigation behavior. (#2934) (Tony Cueva Bravo)
  • Fix system instruction parsing to handle both JSON and plain string formats, improving compatibility with different OpenTelemetry SDKs and preventing export failures due to malformed spans. (#2941) (Tanay)
  • Fix serialization of the tool call type to ensure correct output for any Pydantic serialization method. (#2979) (Vamshi Adimalla)

July

July's release focused on enhancing reliability, safety, and evaluation capabilities for Python developer tools. New deterministic metrics such as AgentLoopDetectionMetric and ToolPermissionMetric enable robust detection of infinite loops and unauthorized tool usage without external dependencies, making them suitable for production environments. Improvements to chatbot evaluation workflows, citation faithfulness checks, and verdict handling further strengthen the accuracy and consistency of conversational AI assessments. The release also brings better error handling, safer JSON serialization, and expanded TypeScript support, alongside workflow automation and documentation refinements to,

New Feature

v4.0.8

  • Add a fully deterministic AgentLoopDetectionMetric that detects infinite loops and cyclical execution patterns in agent traces without requiring an LLM or API key. The metric analyzes tool call repetition, reasoning stagnation, and call graph cycles, returning a weighted score from 0.0 (severe looping) to 1.0 (clean execution). Designed for zero cost and zero latency, it is suitable for production use and includes comprehensive test coverage. (#2645) (Jeel3011)
  • Add an example demonstrating end-to-end evaluation of chatbot responses, including loading questions and responses, generating outputs, and assessing answer relevancy. This example provides a workflow for evaluating conversational AI applications and can be used as a starting point for similar projects. (#2801) (Sunil Gaurav)
  • Add CitationFaithfulnessMetric to evaluate whether each cited passage directly supports its associated claim, enabling detection of citation misattribution in outputs. This metric ensures that claims are attributed to the correct supporting passages, improving the accuracy of citation-based faithfulness checks. Includes documentation and tests for correct and misattributed citations. (#2811) (rsrijith)
  • Add deterministic ToolPermissionMetric to check if agents only call authorized tools based on allowlist and denylist policies, enabling least-privilege enforcement and CI-gateable safety checks without requiring an API key. (#2826) (Raju R)

Improvement

v4.0.8

  • Hardened TurnRelevancy verdict handling by enforcing verdicts as yes or no, matching other metrics. Improved normalization to handle ambiguous responses and prevent misclassification of relevance. Added an evaluation_template parameter for greater flexibility in customizing the evaluation rubric. These changes improve reliability and consistency when using native structured-output models and custom judges. (#2750) (Akshay)
  • Replace all direct uses of json.dumps with a serialize_to_json helper to ensure safer and more consistent JSON serialization across the codebase. This change improves handling of complex objects, nested models, and edge cases such as non-finite floats and non-string dictionary keys, reducing the risk of serialization errors. (#2781) (Vamshi Adimalla)
  • Fix typos in the medical chatbot tutorial pages to improve clarity and readability. (#2800) (Han S. Jung)
  • Add documentation warning that gpt-5 family models without log probability support cannot be used as judge models for G-Eval or ConversationalGEval, and recommend using log-probability-capable models instead. This helps prevent confusion and errors when selecting models for evaluation. (#2805) (Tran Binh Minh)
  • Add support for including a test_run_id when updating the current trace, enabling better tracking and association of traces with specific test runs. (#2816) (Vamshi Adimalla)
  • Honor the DisplayConfig.display_option setting in evaluate() to allow showing only failing or only passing test cases in the terminal output. This makes it easier to focus on relevant results, especially in CI or pre-push hooks where full output can be noisy. When all cases are filtered out, a clear note is shown. The behavior now matches the CLI's --display option. (#2829) (siddharth singh)
  • Fix incorrect markup in documentation to improve formatting and readability. (#2831) (Robert Lehmann)
  • Update release workflows to support new tag-based release process and improve changelog generation. Add a new check to enforce pull request title naming conventions, helping maintain consistency and clarity in PR titles. (#2835) (Vamshi Adimalla)
  • Update the marketplace configuration to reflect the new plugin name and maintain accurate ownership information. (#2852) (Vamshi Adimalla)
  • Add support for specifying a project ID when interacting with datasets and annotations in the TypeScript package. This enables users to associate operations with specific projects, improving organization and access control. The CLI entry point is now properly defined in the package configuration. (#2860) (Vamshi Adimalla)
  • Automate the drafting of release notes by integrating AI into the release workflow, streamlining the process and reducing manual effort. (#2861) (Vamshi Adimalla)
  • Update MCP metrics to use tools_called for improved clarity and reduce type strictness in validation, allowing for more flexible input types. This change makes it easier to integrate with different data sources and reduces errors caused by strict type requirements. (#2864) (Vamshi Adimalla)
  • Fix spelling errors in documentation and code comments to improve clarity and readability. No changes to functionality or APIs. (#2865) (max)
  • Add TypeScript support for MCP metrics to work with both toolsCalled and mcpToolsCalled, and introduce a ToolCall type enum for clearer tool call typing. This improves compatibility and type safety when handling tool calls in metrics and tracing. (#2866) (Vamshi Adimalla)

Bug Fix

v4.0.8

  • Fixes a function signature mismatch in the RAGAS metrics module by updating telemetry calls to include the required in_component argument, ensuring compatibility with the latest telemetry API. (#2568) (Sachin Jha)
  • Fixes a bug where TaskCompletionMetric would fail with a MetricTemplateInterpolationError when running with real test cases by correctly passing a formatted list of called tools to the extract_goal_and_outcome template. The tool names now reliably appear in the rendered prompt, and the no-tools case is handled cleanly. Adds tests to ensure the prompt construction includes the correct tool information and documents the previous failure mode. (#2808) (MinYi Xie)
  • Fixes a TypeError that occurred when hashing ToolCall instances with unhashable output or input parameters. Now, unhashable objects are safely converted to a stable string representation, allowing ToolCall objects to be used in sets and as dictionary keys without errors. Existing behavior for hashable types is preserved. (#2822) (ZentropyLabs.ai)
  • Prevent potential bugs by replacing the mutable list default for turn_params in convert_turn_to_dict with None and a guard, ensuring each call gets a fresh list and avoiding unintended shared state across modules. (#2827) (bono)
  • Fix confusing UnboundLocalError when a document yields zero chunks in context generation, now raising a clear ValueError that explains the issue. This improves error reporting for empty, whitespace-only, or unparseable files, making it easier for users to diagnose chunking problems. (#2849) (Muhammad Hassaan)
  • Restore support for dotted access to deepeval.evaluate.configs and its config classes while keeping deepeval.evaluate callable. This fixes AttributeError when accessing configs via deepeval.evaluate.configs and ensures compatibility with both import styles. (#2851) (ErenAta16)

June

June's release brings major enhancements to both Python and TypeScript interfaces, expanding support for new evaluation models, including Gemini 3 and Claude Opus 4-8, with improved multimodal and structured output capabilities. The update introduces dataset versioning, comprehensive token and cost tracking, and more flexible context generation across multiple files, enabling richer evaluation scenarios and better reproducibility. Governance checks and official baseline designation are now available in the CLI, while numerous documentation improvements and bug fixes ensure a smoother user experience and more reliable integrations. Project organization and test workflows have also been overha

New Feature

v4.0.7

  • Add support for single-turn, multi-turn, arena, and multimodal metrics in the TypeScript interface. This enables more comprehensive evaluation scenarios and allows users to work with a wider range of metric types when using the TypeScript SDK. (#2745) (Vamshi Adimalla)
  • Add support for Gemini 3 series models, including multimodal and structured output capabilities with updated pricing information. (#2774) (Tanay)
  • Add a new governance command to the TypeScript CLI, allowing users to check their project against its governance policy and receive a pass or fail verdict. The CLI provides clear output and exit codes, with options for quiet mode and help. This helps ensure projects comply with organizational governance requirements. (#2779) (Vamshi Adimalla)

v4.0.6

  • Add support for generating and synthesizing contexts that span multiple source files, including new configuration options to control cross-file context creation. Contexts and scenarios now track which source files they use, and prompt generation adapts to multi-file scenarios. This enables more comprehensive synthetic data generation for projects with multiple files. (#2641) (Vamshi Adimalla)
  • Add tracking and reporting of input and output token usage for metrics, allowing users to see token counts alongside evaluation cost. Token usage is now surfaced in metric data and tracing, providing greater transparency into resource consumption during evaluations. (#2717) (Tanay)
  • Add support for marking a test run as the official baseline using the --official CLI flag or the official parameter in the evaluate function. If the --official flag is used without a CONFIDENT_API_KEY environment variable, a warning is shown and the run is not marked as official. This helps users clearly designate baseline runs for Confident AI integration. (#2732) (Vamshi Adimalla)
  • Add support for new evaluation models in both Python and TypeScript, including improved handling of model configuration and cost calculation. The update introduces a shared gateway model base for Python, refines temperature parameter logic for Anthropic models, and adds OpenAI-compatible model support in TypeScript. These changes make it easier to use a wider range of providers and improve reliability when working with different LLM gateways. (#2738) (Vamshi Adimalla)

v4.0.5

  • Add support for dataset versioning, allowing users to create, retrieve, and manage multiple versions of a dataset. Users can now specify a version when pushing or pulling datasets, and list all available versions for a given dataset alias. This enables better tracking and reproducibility of dataset changes over time. (#2697) (Vamshi Adimalla)
  • Add support for the claude-opus-4-8 model with multimodal and structured output capabilities, as well as JSON support. (#2698) (Vamshi Adimalla)

Improvement

v4.0.7

  • Improve ContextualPrecisionMetric by grouping retrieved context data from the same source, reducing redundant LLM calls and providing better context to the judge model. Fix the weighted cumulative precision formula to align with documentation, ensuring more accurate scoring. (#2743) (rello-code)
  • Move metric template tests to a new workflow file and update documentation to reflect the new test location. This improves test organization and ensures that template synchronization checks are run independently. (#2768) (Vamshi Adimalla)
  • Improve serialization by using a custom Pydantic-compatible serializer as the default for all json.dumps calls, ensuring better handling of complex objects and more consistent JSON output across integrations and metrics. (#2778) (Vamshi Adimalla)

v4.0.6

  • Move the TypeScript package into a dedicated typescript folder, organizing related source code, configuration, and examples for improved project structure and maintainability. Adds separate linting and testing workflows for the TypeScript codebase. (#2718) (Vamshi Adimalla)
  • Update documentation to explain how to mark a test run as the official baseline on Confident AI using the --official flag or official=True parameter. Clarifies usage in both CLI and Python API, and notes the requirement for a Confident AI API key. (#2740) (Vamshi Adimalla)

v4.0.5

  • Fix broken code examples and clarify method requirements in the custom LLMs guide, ensuring users can successfully implement non-OpenAI providers without encountering errors. The guide now accurately reflects the expected method signatures and highlights common pitfalls, improving reliability for users following these instructions. (#2677) (Liz Zhang)
  • Corrected documentation to state that telemetry is handled by PostHog, not Sentry, and clarified the distinction between always-on telemetry and opt-in error reporting. Added a detailed list of data collected by PostHog to help users make informed opt-out decisions. (#2680) (Aadhar Agarwal)
  • Fix typos in tutorials and documentation to improve clarity and accuracy. No changes to tool behavior. (#2689) (Jsu)
  • Update documentation to include the latest Anthropic models such as claude-opus-4-8, claude-opus-4-7, claude-opus-4-5, claude-sonnet-4-6, claude-sonnet-4-5, and claude-haiku-4-5. (#2699) (Vamshi Adimalla)
  • Allow deepeval test run to accept any Python file, not just those starting with test_, aligning behavior with pytest and removing unnecessary filename restrictions. File and directory validation now defers to pytest, ensuring consistency and reducing friction for users with custom file naming. (#2700) (Alex Maggioni)
  • Fix word order in the retrieval hyperparameter note in the RAG Evaluation guide for clearer documentation. (#2709) (Tai An)
  • Expanded documentation to explain how users of custom models can enable GEval weighted summation, including a code example for implementing the required method. (#2713) (nuthalapativarun)
  • Fix several typos in the README and metrics introduction documentation to improve clarity and accuracy. No changes to code or functionality. (#2714) (Hibou04)

Bug Fix

v4.0.7

  • Fix cost calculation and configuration for Gemini models, allowing users to specify input and output token costs via environment variables or constructor parameters. Cost reporting now accurately reflects custom values, and documentation has been updated to describe the new options. (#2724) (Vamshi Adimalla)
  • Fix incorrect URL redirection in the getting started documentation so that the introduction link now points to the correct page. (#2747) (muller-j)
  • Fix AmazonBedrockModel to correctly store and use user-supplied per-token costs, ensuring that custom models with explicit costs return accurate values from calculate_cost. Registered models and other providers are unaffected. Adds a regression test to prevent similar issues. (#2753) (Lucas Kim)
  • Fix incorrect import path that caused the official tutorial to fail with a ModuleNotFoundError when running tests, ensuring the CLI works as expected. (#2755) (Andrew Tennikoff)
  • Fix cost and token accumulation for leaf VerdictNode to ensure accurate tracking of evaluation resources during metric execution. This prevents underreporting of costs and tokens when using DAG-based metrics. (#2756) (Tanay)
  • Fixes an issue where agent spans created via the pydantic-ai integration did not populate the tools_called field, which previously caused crashes in tool-aware evaluation and prevented correct metric computation. Tool calls are now correctly extracted from message history and included in agent spans, enabling proper evaluation of pydantic-ai agents. No schema or API changes were made. (#2766) (bonpiedlaroute)
  • Fix LangGraph integration in the TypeScript callback handler to ensure correct tracing and context management when running under a LangGraph server deployment. The handler now properly manages per-request trace creation, preventing issues when the handler is reused across multiple server requests. Documentation has been updated to clarify how to attach the callback for server deployments and the implications for trace attributes. (#2777) (Vamshi Adimalla)

v4.0.6

  • Fix integration tests to properly handle missing API keys, ensuring tests are skipped rather than failing when required keys are not set. Improve context restoration in LangChain integration to prevent incorrect span context after runs. Remove hardcoded test API keys from test initialization for better security and reliability. (#2719) (Vamshi Adimalla)
  • Fix crash in TurnFaithfulnessMetric when processing assistant turns with no factual claims, ensuring it returns a valid score and reason instead of raising a TypeError. Both synchronous and asynchronous variants now handle empty verdicts gracefully. Added unit tests to cover these cases and prevent regressions. (#2720) (Aarkin Karnik)
  • Fix EvaluationDataset.save_as to correctly serialize RetrievedContextData objects in retrieval_context, preventing crashes when saving datasets in json, csv, or jsonl formats. Now, all retrieval_context items are routed through the model's serializer, ensuring consistent output and compatibility with existing serialization behavior. (#2721) (Joeseph Grey)
  • Fix GeminiModel to correctly report evaluation costs based on actual token usage and pricing, instead of always returning zero. This ensures accurate cost reporting in the console, EvaluationCost, and dashboards, and aligns Gemini cost calculations with other providers. Tests have been updated to verify correct cost computation and prevent regressions. (#2722) (Aarkin Karnik)
  • Propagate all non-zero pytest exit codes to ensure test failures and errors are correctly reported in CI environments. (#2735) (Vamshi Adimalla)

v4.0.5

  • Fixes a bug in the async TurnContextualRecallMetric where the expected outcome was not correctly passed, ensuring retrieval context is properly evaluated. Adds a regression test using a fake LLM to catch this issue without requiring external API calls. This prevents valid conversational retrievals from being incorrectly scored as zero. (#2694) (Ritwij Aryan Parmar)
  • Fixes an issue where the trimAndLoadJson function could corrupt valid JSON string values containing sequences like ', ]' or ', }'. Now, valid JSON is parsed directly without modification, and the trailing comma cleanup is only applied as a fallback if parsing fails. This ensures that string values are preserved and only malformed JSON with genuine trailing commas is altered. Additional tests confirm that valid JSON round-trips unchanged and errors are raised for genuinely invalid input. (#2701) (Kyle)
  • Fix variable name mismatch in the metrics introduction documentation to prevent NameError when copying code examples. (#2703) (nuthalapativarun)
  • Fixes an issue where TurnContextualRecallMetric returned incorrect scores in async mode by ensuring the expected outcome is passed correctly. Async and sync measurement paths now behave consistently. Adds a regression test to prevent similar issues in the future. (#2708) (Parafee41)

May

May's release brings significant enhancements to multimodal tracing, enabling seamless analysis of workflows involving images and PDFs. The conversation simulator now supports flexible simulation graphs, offering advanced control over conversation flows and improved extensibility. Integrations have expanded to include Google ADK and Strands, with robust OpenTelemetry-based tracing and comprehensive documentation updates. Prompt optimization algorithms have been streamlined for efficiency, and new features such as a CLI trace inspector and improved evaluation reporting enhance usability. Numerous fixes and test improvements ensure greater reliability, traceability, and performance across the

New Feature

v4.0.3

  • Add support for multimodal traces, including both images and PDFs, for LangChain and native traces. Traces now capture and expose attachments such as images and PDFs, making it easier to analyze and debug workflows involving multimodal content. Improved detection and handling of multimodal placeholders throughout the codebase. (#2672) (Vamshi Adimalla)
  • Add support for simulation graphs in the conversation simulator, allowing users to define flexible and customizable conversation flows using SimulationNode and related classes. The simulator now accepts a simulation_graph parameter, enabling advanced control over user turn generation and conversation structure. The previous controller parameter is now deprecated in favor of stopping_controller, with warnings issued for deprecated usage. This update improves extensibility and user control over AI. (#2678) (Jeffrey Ip)
  • Add support for pulling metric definitions from the server for both GEval and ConversationalGEval metrics. Users can now retrieve and instantiate metrics by name, ensuring local metric objects are synchronized with server-side definitions. This improves workflow flexibility and consistency when working with shared or remote metrics. (#2682) (Vamshi Adimalla)
  • Add agent skills for both Cursor and Claude, enabling integration of DeepEval evaluation, tracing, and reporting capabilities into AI applications using these platforms. Introduce a new skill for exporting raw OpenTelemetry traces from AI applications to Confident AI's Observatory without requiring the DeepEval package, supporting vendor-neutral observability. Update documentation to clarify when to use each skill and improve guidance for instrumenting AI applications. (#2684) (Vamshi Adimalla)
  • Add a separate skill for DeepEval native tracing, allowing users to instrument AI applications with detailed span types, tags, and metadata for improved trace visibility in Confident AI. This separation clarifies when to use DeepEval's own tracing versus OpenTelemetry or evaluation features, making it easier to choose the right instrumentation approach for Python AI apps. (#2685) (Vamshi Adimalla)

v4.0.2

  • Add support for using OpenRouterModel as a native evaluation provider in metric utilities. OpenRouter is now recognized in model initialization and detection, enabling seamless integration for both synchronous and asynchronous evaluation. Cost tracking and automated provider selection are also supported. (#2632) (Djalal-H)
  • Add google-adk and openinference-instrumentation-google-adk packages to the dependencies, enabling integration with Google ADK for supported Python versions. (#2650) (Vamshi Adimalla)
  • Add JSON schemas for Google ADK integration tests, enabling structured validation and improved test coverage for Google ADK-related features. (#2660) (Vamshi Adimalla)
  • Add integration with the Strands framework, enabling OpenTelemetry-based tracing and span collection for Strands agents. This integration supports full feature parity with other OTel-mode integrations, including synchronized trace contexts and span-level metric collection. Users can now instrument Strands agents for detailed observability and evaluation workflows. (#2661) (Vamshi Adimalla)
  • Add integration and documentation for Google ADK, enabling users to instrument and trace LLM agents built with Google ADK in their workflows. (#2662) (Jeffrey Ip)
  • Add a new inspect command to the CLI that opens a local trace view for saved test runs, allowing users to visually inspect evaluation traces in a terminal-based interface. The command supports specifying a file or folder, and provides helpful install hints if required dependencies are missing. (#2666) (Jeffrey Ip)

Improvement

v4.0.2

  • Improve prompt optimization by updating the COPRO algorithm to use a more efficient and streamlined approach, reducing complexity and enhancing maintainability. The new implementation simplifies configuration, improves reproducibility with better random state handling, and clarifies the optimization process for users. (#2639) (Vamshi Adimalla)
  • Fix minor typos in documentation to improve clarity and accuracy. (#2642) (yuri)
  • Add support for Apple Silicon MPS devices in BERTScorer, allowing Mac users with compatible GPUs to utilize hardware acceleration instead of defaulting to CPU. Device selection now prioritizes CUDA, then MPS, then CPU, improving performance on Apple Silicon Macs. (#2646) (Boao(Mark) Dong)
  • Improve documentation for installing and using skills with Claude.ai and the local CLI, and consolidate workflow instructions for the DeepEval skill into a single location. (#2649) (Vamshi Adimalla)
  • Add new tests for Google ADK integration covering multiple tool usage, parallel tool calls, and span-level configuration using per-call context managers. These tests ensure that metric collection and trace context features work as expected when invoking agents and tools, improving reliability and coverage of the integration. (#2654) (Jeffrey Ip)
  • Add provider and integration fields to tracing types, allowing more detailed tracking of model providers and integration sources in tracing data. (#2655) (Vamshi Adimalla)
  • Add documentation and UI components for Google ADK and improve integration tags display in the docs. The updated documentation provides clearer guidance on using AgentCore with deepeval, including step-by-step instructions, code examples, and visual enhancements for integration features such as OTel instrumentation, CI/CD evals, and traceability. (#2656) (Jeffrey Ip)
  • Add support for passing provider and integration fields across all integrations, ensuring that tracing and span data include this metadata. Update tests to validate the presence of these fields, improving traceability and consistency in integration outputs. (#2657) (Vamshi Adimalla)
  • Add regression tests to ensure the test-run OpenTelemetry tracer does not override the host application's global tracer provider, preventing unintended side effects when integrating with other telemetry setups. (#2658) (mindbomber)
  • Improve console output for evaluation results by providing a more detailed and visually enhanced report, including support for HTML and Markdown exports. Users can now choose to truncate passing cases in the terminal and select the output file type for saved reports. This makes it easier to review and share evaluation outcomes. (#2663) (Vamshi Adimalla)
  • Add documentation for the SIMBA and COPRO prompt optimization algorithms, including detailed usage instructions, configuration options, and guidance on when to use each method. Users can now find comprehensive guides for both algorithms in the documentation, making it easier to understand their features and integrate them into prompt optimization workflows. (#2664) (Vamshi Adimalla)
  • Update dependencies in the lock file to include new and updated packages such as linkify-it-py, mdit-py-plugins, textual, and uc-micro-py. This ensures compatibility and provides the latest features and fixes from these libraries. (#2667) (Vamshi Adimalla)
  • Improve the contextual precision metric to group retrieval context entries by their source when using RetrievedContextData objects. This allows for more accurate evaluation of retrieval performance when multiple contexts originate from the same source. Documentation has been updated to clarify the accepted types for retrieval_context and the grouping behavior. (#2669) (Vamshi Adimalla)
  • Add an automated workflow to generate and extract release notes for each new version, streamlining the release process and ensuring consistent changelog updates. (#2671) (Vamshi Adimalla)

Bug Fix

v4.0.2

  • Prevent crashes in trimAndLoadJson when input is None by adding a guard and raising a descriptive error, ensuring consistent error handling and clearer feedback when the evaluation LLM returns invalid output. (#2620) (bono)
  • Return a perfect score of 1.0 for knowledge retention when no verdicts exist, ensuring that short conversations with nothing to forget are correctly evaluated as full retention. This prevents contradictory results where the reason indicates all knowledge is retained but the score was previously 0. (#2636) (Nguyen D. Mau)
  • Fix FaithfulnessMetric to include ambiguous claims as contradictions when penalize_ambiguous_claims is enabled, ensuring the generated reason accurately reflects penalizations for ambiguous claims. (#2644) (miafig)
  • Fix incorrect import path for DeepAcyclicGraph in the DAG metric documentation to prevent ModuleNotFoundError and ensure consistency with other examples. (#2647) (Jayden Chua)
  • Fixes integrations to ensure trace capture failures are detected and reported, preventing silent test passes when trace data is missing. Improves CLI startup performance by lazily loading synthesizer modules only when needed. Adds an instrument function to simplify OpenTelemetry backend setup. Ensures all root spans are walked for accurate trace hydration, even on errored traces, improving dashboard diagnostics. (#2648) (Jeffrey Ip)
  • Fix integration tests for OpenAI agents and LangChain to ensure compatibility and correct behavior. Updates improve the handling of span types and agent extraction, leading to more reliable test outcomes and better traceability in integrations. (#2651) (Vamshi Adimalla)
  • Fix trace metadata handling so that user-configured trace metadata is correctly forwarded to the API trace, preventing it from being overwritten by evaluation-specific metadata. Update documentation and integration patterns to clarify the separation between trace-level and test-case metadata, and improve AgentCore integration to follow the recommended OpenTelemetry pattern for consistent metric collection and context propagation. (#2653) (Jeffrey Ip)

April

April's release focused on streamlining configuration and improving model support, with legacy options removed and new features added for both OpenAI and Anthropic models, including multimodal and structured output capabilities. The documentation site was overhauled for better usability, and tracing was enhanced with new guides, expanded metadata, and improved integration across workflows. The release also introduced a powerful generate CLI for synthetic goldens, prompt branch management, and a skills-compatible deepeval integration, while addressing numerous bugs and refining test infrastructure for more reliable and insightful evaluation.

Breaking Change

v3.9.9

  • Remove the legacy API_KEY alias and require CONFIDENT_API_KEY for Confident uploads. Update dataset loading to use metadata instead of additional_metadata, and refresh docs/examples to use SingleTurnParams for GEval evaluation parameters. (#2635) (Jeffrey Ip)

v3.9.8

  • Remove the observed_callback hook from assert_test and rely on the active trace when asserting against a golden. Add results_folder/results_subfolder options to control where full structured test-run JSON is saved, and deprecate per-result .log output. (#2622) (Jeffrey Ip)
  • Remove the metric logging manager and related configuration options, simplifying debug settings and API endpoints. Update CI to run the simulator test suite and reorganize conversation simulator tests for the new layout. (#2629) (Jeffrey Ip)

New Feature

v3.9.9

  • Add a generate CLI command to create synthetic goldens from documents, contexts, scratch prompts, or existing goldens, with configurable output format, concurrency, and styling options. (#2633) (Jeffrey Ip)
  • Add a Cursor/skills-compatible deepeval skill with templates and guidance for generating datasets, creating pytest eval suites, enabling tracing, and iterating on evaluation failures. (#2634) (Jeffrey Ip)
  • Add metadata and tags support to both SingleTurnParams and MultiTurnParams, making it easier to pass custom context through single-turn and conversational evaluation workflows. (#2635) (Jeffrey Ip)

v3.9.8

  • Add support for the claude-opus-4-7 model, including multimodal inputs, structured outputs, and JSON mode, with updated pricing metadata. (#2617) (Tanay)
  • Add a conversation simulator controller API with proceed()/end() decisions, plus a public SimulationTemplate. Update and expand simulator tests and CI coverage, including safer defaults when controllers return None or unexpected values. (#2628) (Jeffrey Ip)

v3.9.6

  • Add support for the gpt-5.4-mini model. Metrics that rely on log probabilities now detect when the model doesn’t support them and avoid failing with unexpected errors. (#2603) (Tanay)

v3.9.5

  • Add support for extracting confident.trace.test_case_id in ConfidentSpanExporter so OTel-exported traces can propagate testCaseId and be linked to the right test case instead of always being null. (#2570) (Alex Maggioni)
  • Add prompt branch support, including pushing to a specific branch and listing, creating, renaming, and deleting branches. Cache and commit lookups can now be scoped by branch to keep versions organized. (#2583) (Vamshi Adimalla)

Improvement

v3.9.9

  • Improve OpenAI defaults by switching the default GPT model to gpt-5.4 when no model is configured. Add model metadata for gpt-5.4 (and its snapshot alias) and update JSON output support flags for relevant models. (#2630) (Jeffrey Ip)
  • Improve changelog and docs parsing by supporting React-style comment markers for release note markers, PR tags, and changelog-ignore blocks, while remaining compatible with the legacy HTML comment format. (#2631) (Jeffrey Ip)

v3.9.8

  • Improve documentation site by migrating to a new Next.js-based setup with updated layouts and built-in search, along with refreshed docs tooling and ignores for generated build artifacts. (#2624) (Jeffrey Ip)
  • Improve docs site link previews and layout by adding a default Open Graph image, tightening homepage spacing, and fixing overflow/scrolling behavior in code and terminal demo blocks. (#2627) (Jeffrey Ip)

v3.9.7

  • Improve telemetry dependency compatibility by using PostHog 7.x on Python 3.10+ while keeping PostHog 5–6 on Python 3.9 via environment markers. (#2605) (Manoj Kumar Nagabandi)

v3.9.6

  • Add new tracing guides for multi-turn chatbots, RAG flows, and AI agents, including examples for grouping turns with thread_id and instrumenting spans for better end-to-end observability. (#2581) (Vamshi Adimalla)
  • Add a manual GitHub Actions workflow to generate changelog updates for a given year or tag range and open an update pull request automatically. (#2588) (Vamshi Adimalla)
  • Add optional internal tracing for metric and model methods called inside @observe spans, controlled by CONFIDENT_TRACE_INTERNAL. When enabled, key LLM generation methods and metric execution paths are captured with more detailed nested spans; when disabled, this extra instrumentation is skipped to reduce overhead. (#2589) (Kritin Vongthongsri)

v3.9.5

  • Add optional turn_id and test_case_id fields to tracing across supported integrations, and include them in exported trace payloads for easier correlation of multi-turn runs. (#2576) (Kritin Vongthongsri)

Bug Fix

v3.9.8

  • Fix multi-turn Pydantic trace input to use the most recent user message instead of the first. This prevents follow-up questions from incorrectly showing the initial question as the trace input. (#2614) (Brian Romain)
  • Fix a KeyError in MLLMImage.parse_multimodal_string when parsing [DEEPEVAL:IMG:<id>] markers for images that aren’t already registered. Newly created images are now kept alive for the caller so registry lookups don’t fail. (#2615) (Tanay)
  • Fix Anthropic Opus 4.7 requests by omitting temperature when the model does not support it, preventing API errors in both sync and async generation. (#2618) (Tanay)
  • Prevent evals_iterator runs from silently doing nothing by raising a clear error when no metrics are declared at any level. This avoids misleading end-of-run messages and makes missing metric configuration easier to diagnose. (#2621) (Jeffrey Ip)

v3.9.7

  • Fix loading single-turn golden tool calls from CSV by parsing tools_called and expected_tools as JSON objects instead of splitting by a delimiter, matching the format produced by save_as. This prevents errors and ensures tool call goldens round-trip correctly through CSV. (#2565) (Sean Kelley)
  • Add missing Anthropic model entries for Claude Opus 4.6 and Sonnet 4.6, including dated IDs and short aliases. Fix Opus 4.5 pricing so cost reports are no longer inflated. Restore cost tracking for default *-latest models by registering their IDs so require_costs() no longer falls back to None. (#2584) (Ajay Sai Reddy Desireddy)
  • Fix conversion of conversational goldens to preserve expected_outcome, preventing metrics that rely on it from failing validation or skipping evaluation after conversion. (#2598) (aerosta)
  • Fix OpenAI tracing spans for newer gpt-5.x/Responses API models to correctly record input/output token counts and populate per-token cost data. This prevents None values when instrumenting an OpenAI client via patch_openai_client. (#2601) (tiffanychum)
  • Fix PydanticAI tracing integrations by correctly classifying agent vs LLM spans and preventing mislabeling when agent attributes are present. Improve message normalization across instrumentation versions and ensure trace context is properly reset after a trace ends. (#2606) (Vamshi Adimalla)
  • Fix schema construction for structured outputs by correctly unwrapping Optional[...] types and detecting nested Pydantic models through full inheritance. This prevents Optional[List[int]] from being misclassified as STRING and ensures derived BaseModel types are recognized as OBJECT. (#2611) (SamSi0322)
  • Fix _mcp_interaction detection so MCP usage is correctly recognized under Pydantic v2. This prevents MCP-related metrics from returning near-zero scores when tools, resources, or prompts were actually called. (#2612) (SamSi0322)

v3.9.5

  • Fix type annotations for model and using_native_model in base metric classes by making them proper optional fields with None defaults, improving static type checking and reducing annotation-related errors. (#2574) (Tommy Beadle)
  • Fix docs structured data by removing the Product schema for metric pages and generating only Article schema. This avoids incorrect organization and product metadata in the rendered schema output. (#2577) (Vamshi Adimalla)

March

March brought major improvements to observability and integration, with AgentCore and OpenInference now supporting robust OpenTelemetry instrumentation, span classification, and trace export via OTLP. Enhanced configuration options and clearer error handling make it easier to connect with a variety of providers, including Amazon Bedrock and Azure OpenAI, while new documentation and test coverage help users adopt these features confidently. Numerous bug fixes address trace reliability, async evaluation, memory leaks, and provider-specific quirks, resulting in more stable and accurate metrics across diverse workflows. Code formatting and validation improvements further streamline development,,

New Feature

v3.9.1

  • Add AgentCore integration with OpenTelemetry instrumentation, including span classification and message extraction for agent, tool, and LLM traces. Support exporting telemetry via OTLP with configurable metadata, tags, and metric collection, and provide a test mode exporter for local validation. (#2534) (Vamshi Adimalla)
  • Add OpenInference integration to intercept OpenTelemetry spans, extract LLM/agent/tool inputs and outputs, and export traces via OTLP. Provides configurable metadata, tags, and metric collection, and surfaces clear errors when required OpenTelemetry deps or CONFIDENT_API_KEY are missing. (#2555) (Vamshi Adimalla)

v3.8.9

  • Add custom_column_key_values to LLMTestCase to store custom metadata as a Dict[str, str]. Accept both custom_column_key_values and customColumnKeyValues on input and serialize as customColumnKeyValues, with type validation for safer usage. (#2530) (Brian Romain)

Improvement

v3.9.3

  • Improve code formatting consistency by reformatting the codebase with the latest Black rules, reducing lint noise and keeping style checks stable across environments. (#2567) (Vamshi Adimalla)
  • Add documentation describing the updated Amazon Bedrock integration behavior, helping users configure and use Bedrock correctly after recent changes. (#2571) (Vamshi Adimalla)

v3.9.1

  • Add AWS AgentCore integration documentation and CI coverage, and improve span extraction and test-mode handling. Also allow tuning OTLP batch exporter settings to better control export timing and batch size. (#2544) (Vamshi Adimalla)
  • Fix AgentCore tracing to capture agent and trace input/output reliably, avoid duplicate traces during evaluation, and recognize additional GenAI span attributes. Also simplify instrument_agentcore by removing OTEL exporter tuning options and update docs to use evals_iterator() for end-to-end eval runs. (#2545) (Vamshi Adimalla)
  • Support selecting the evaluation provider via environment variables and passing a model name as a string when initializing metrics. This makes it easier to switch between OpenAI, Anthropic, Gemini, Azure, and local backends without changing code. (#2550) (Vamshi Adimalla)

v3.8.9

  • Support setting metric_collection on the active trace and span via the update helpers. This makes metric collection configuration consistent when updating an in-progress trace rather than only at trace creation. (#2532) (Vamshi Adimalla)

Bug Fix

v3.9.3

  • Fix chunk-size validation to treat collection.count() as chunk count rather than token count, preventing incorrect errors when generating contexts. Improve the guidance in the exception message with clearer suggestions to reduce chunk_size and chunk_overlap. (#2468) (Xuan-Phung Pham)
  • Fix initialize_model() to recognize Amazon Bedrock configuration so USE_AWS_BEDROCK_MODEL=YES no longer falls back to the default GPT model. This prevents silently using the wrong provider when Bedrock is intended. (#2537) (Parafee41)
  • Fix LangChain/LangGraph token usage extraction by reading usage_metadata with a fallback to legacy response_metadata, improving callback accuracy across versions. Improve test stability by retrying flaky Confident and integration tests and updating integration dependencies and fixtures. (#2557) (Vamshi Adimalla)
  • Fix async trace evaluation so per-trace metrics aren’t shared across concurrent tasks. This prevents concurrent runs from overwriting score, reason, and success, eliminating timing-dependent and inconsistent results. (#2559) (aerosta)
  • Fix pydantic-ai tracing so thread_id, name, and metadata set via the current trace context are exported on span start. Falls back to settings for compatibility and merges settings metadata with per-request metadata. (#2563) (Oluwanifemi Adeyemi)

v3.9.1

  • Fix Azure OpenAI keyless authentication by deferring credential checks to the OpenAI SDK. Only fail fast when an explicit credential is provided but empty, while preserving key-based auth and handling both SecretStr and string credentials consistently. (#2464) (ppon1086)
  • Fix a Pydantic ValidationError in KnowledgeRetentionMetric._extract_knowledges by correctly unpacking LLM response dicts when creating Knowledge objects, preventing double-wrapping and improving validation reliability. (#2513) (Diego GΓ³mez Moreno)
  • Fix evaluate() in CLI runs to stop resetting the test run manager so results from multiple files are accumulated and reported together. Add a skip_reset option for manual control outside CLI mode. Ensure test case order values are always unique to prevent earlier results being overwritten or shown as skipped. (#2529) (Alex Maggioni)
  • Fix CrewAI tool tracing when events arrive out of order from the thread pool. Tool spans are now created and closed reliably using finished-event data, with corrected timestamps and consistent propagation of called tools to the parent span. (#2547) (Vamshi Adimalla)
  • Fix ConversationalGEval to make top_logprobs configurable. Add a top_logprobs parameter to the initializer (default 20) and use it in both sync and async execution paths instead of a hardcoded value. (#2549) (Szymon Cogiel)
  • Fix a memory leak when processing many multimodal test cases by storing _MLLM_IMAGE_REGISTRY in a weakref.WeakValueDictionary. Unreferenced MLLMImage instances are now garbage-collected automatically, preventing unbounded memory growth in large batch runs. (#2551) (eason)
  • Fix metric cache loading to ignore incomplete cached entries and fall back to recomputing when no score is available. Progress reporting now updates correctly when cached results are used. (#2552) (Konstantin)

v3.8.9

  • Fix generator tracing so observers record the final yielded item when a generator finishes without returning a value, improving captured outputs for streaming workflows. (#2514) (Vamshi Adimalla)
  • Fix FilterTemplate.evaluate_context examples by removing duplicate contexts and replacing them with distinct ones. Each example now has a unique input/output pairing, avoiding repeated contexts with different scores. (#2518) (Fiza Mukhtar)
  • Fix ContextConstructionConfig.critic_model defaulting to a new model when unset. It now falls back to the model passed to Synthesizer, so you only need to specify a custom model once when generating goldens from docs. (#2520) (Br1an)
  • Fix HumanEval evaluation so test assertions run against the generated function by executing the function and tests in the same sandboxed context. Also treat runtime exceptions as failures and expand allowed builtins needed by common HumanEval test cases. (#2521) (Br1an)
  • Fix temp ChromaDB directory cleanup on Windows by stopping the client system before calling shutil.rmtree. This releases open SQLite file handles and prevents PermissionError: [WinError 32] during teardown, with retries kept as a fallback. (#2522) (Br1an)
  • Fix calculate_weighted_summed_score to avoid ZeroDivisionError when all token logprobs are filtered out and the probability sum is 0. When no tokens survive filtering, it now falls back to the raw score instead of failing. (#2524) (VENKATA PRANAY BATHINI)
  • Fix the Goal Accuracy Score equation to remove a circular dependency. The formula now matches the implementation by averaging Goal Evaluation Score and Plan Evaluation Score as two distinct components. (#2526) (JevDev2304)

February

February's release focused on enhancing tool and prompt support, with improvements to schema handling and prompt versioning that make it easier to manage and call tools programmatically. Dependency management was streamlined by removing unnecessary packages and improving compatibility across platforms. Several bug fixes addressed issues in tracing, prompt metadata, and documentation, while new features like AU data region support and more robust Windows cleanup further improved usability and reliability. Overall, this month brought greater consistency, flexibility, and stability to the developer experience.

New Feature

v3.8.4

  • Add support for overriding the default cache directory via an environment variable, allowing you to relocate cached files without changing code. (#2455) (vection)
  • Add tool support to pulled and cached prompts, exposing any returned tools and converting their structured fields into a JSON Schema input_schema for easier function/tool calling. (#2466) (Vamshi Adimalla)
  • Add tool support to prompt push and update, including optional tools payloads. Improve tool schema handling by reusing output schema conversion and generating JSON Schema input parameters consistently, even for empty schemas. (#2474) (Vamshi Adimalla)

Improvement

v3.8.7

  • Fix minor typos in the Tool Correctness metric documentation, including list formatting and a missing final newline for cleaner rendering. (#2504) (nikkie)

v3.8.5

  • Improve prompt handling after the API migration by switching prompt identity and caching to use commit hashes and adding support for prompt commits endpoints. This helps prompts and logged hyperparameters stay consistent when versions change, and preserves tools and schema data when pulling from cache. (#2475) (Vamshi Adimalla)
  • Remove the opentelemetry-exporter-otlp-proto-grpc dependency from the default install to reduce required packages and keep OpenTelemetry exporter tooling out of core installs. (#2477) (Tommy Beadle)
  • Improve dependency compatibility by allowing Click 8.3.x (click < 8.4.0). Also adjust Linux-only pysqlite3-binary handling to prevent install failures on other platforms. (#2486) (Muhammad Faizan)
  • Improve prompt logging in traces by recording prompt alias, commit hash, label, and version as first-class span fields across integrations and the OTEL exporter. (#2487) (Vamshi Adimalla)
  • Add support for the AU data region for Confident AI requests. The CLI can now set region to AU, and API routing will automatically use AU endpoints when your API key starts with confident_au_. (#2494) (Vamshi Adimalla)

Bug Fix

v3.8.7

  • Fix Azure OpenAI requests to omit temperature for reasoning models that don’t support it, preventing Azure API errors. Validation now allows temperature=None, and defaults remain unchanged for standard models. (#2491) (aerosta)
  • Fix tests by updating the hardcoded valid trace UUID used in annotation test fixtures. (#2505) (Vamshi Adimalla)

v3.8.8

  • Fix the @observe decorator to capture the final return value from synchronous generator functions, not just yielded items. Also ensure the observer is closed reliably on normal completion, GeneratorExit, and errors. (#2509) (Vamshi Adimalla)

v3.8.6

  • Fix offline trace/span evaluation requests by sending the correct endpoints and parameters, and add overwrite_metrics support. Also allow passing chatbot_role when evaluating a thread, and avoid printing tool calls when none are present. (#2498) (Vamshi Adimalla)

v3.8.5

  • Fix the TaskCompletionMetric docs example to use goldens instead of golden, preventing a NameError when iterating over dataset.evals_iterator. (#2454) (Himanshu Kumar Singh)
  • Fix a_generate_with_schema_and_extract to handle models that return (result, cost) tuples. It now accrues cost when supported and extracts the actual result so downstream processing works without tuple parsing errors. (#2470) (Angelen)
  • Fix AmazonBedrockModel raising AccessDeniedException during async evaluations when AWS credentials are valid. Improves async-safe aiobotocore session and credential handling to prevent loss under concurrency while keeping sync behavior unchanged. (#2471) (Fiza Mukhtar)
  • Fix the conversation completeness prompt to better extract user intentions by separating multiple tasks per turn instead of summarizing them into one intention. (#2478) (Vamshi Adimalla)
  • Fix the knowledge retention extraction template to wrap extracted fields under a top-level data object, matching the expected JSON output format and improving parsing reliability. (#2479) (Vamshi Adimalla)
  • Fix OpenTelemetry span attribute setting to avoid sending None values for prompt metadata, reducing invalid or noisy telemetry attributes. (#2488) (Vamshi Adimalla)
  • Fix tracing prompt metadata conversion so prompt_alias, commit hash, label, and version are set consistently and don’t get overwritten by empty prompt objects. Improve prompt tests to avoid cache and alias collisions, making pull and cache behavior more reliable. (#2489) (Vamshi Adimalla)
  • Fix CrewAI tracing to capture prompt metadata and expected outputs on LLM spans, improve tool-span detection, and make tool completion more reliable when duplicate events or key mismatches occur. Also broaden metric lookup to support both underscored and public attribute names. (#2490) (Vamshi Adimalla)
  • Fix synthesizer crashes when include_expected_output=False and when max_quality_retries=0, preventing AttributeError and UnboundLocalError. Correct goldens generation so evolutions_used metadata no longer leaks across iterations. Add tests covering these scenarios. (#2493) (aerosta)
  • Fix a crash in deepeval view after login when telemetry doesn’t create a span. upload_and_open_link() now treats the span as optional and only sets attributes when it exists, so the command completes instead of raising an AttributeError. (#2496) (Jeremy Johnson)

Security

v3.8.5

  • Improve cleanup on Windows by updating safe_rmtree to use subprocess.run with argument lists instead of os.system. This handles paths with spaces more reliably and reduces the risk of command injection, making directory removal more robust across environments. (#2484) (Rin)

January

January saw major improvements in model integration, authentication, and developer experience. The release introduced OpenRouter support with flexible model APIs, enhanced Azure OpenAI authentication, and explicit control over Vertex AI and Gemini clients. Changelog generation and documentation were overhauled for clarity and consistency, while tracing and metric collection became more robust across LangChain, LangGraph, and Pydantic AI integrations. Numerous bug fixes addressed edge cases in evaluation, tracing, and API routing, resulting in a more reliable and user-friendly toolset.

New Feature

v3.8.1

  • Add support for OpenRouter with an OpenAI-compatible model API and dynamic model names. Support structured outputs, configurable retries, and custom headers like HTTP-Referer and X-Title. Allow user-provided pricing with fallback to provider-reported pricing. (#2314) (Wang Junwei)
  • Add an automated changelog generator that builds ClickHouse-style release notes from git tags, with optional GitHub and AI enrichment. Backfill the docs changelog for 2025 to match the new year/month/category layout. (#2403) (Trevor Wilson)
  • Add upload() support for GEval and ConversationalGEval to send metric definitions (criteria/steps, required parameters, rubric, multi-turn) to Confident AI and store the returned metric id. (#2419) (Vamshi Adimalla)
  • Add a use_vertexai option to explicitly choose between Vertex AI and Gemini API-key clients when creating GeminiModel. This overrides the GOOGLE_GENAI_USE_VERTEXAI setting, including forcing False to avoid Vertex AI even if project/location are set. (#2436) (Trevor Wilson)
  • Add support for authenticating Azure OpenAI models using azure_ad_token or an azure_ad_token_provider, so you can use Azure AD credentials instead of an API key when desired. (#2448) (Vamshi Adimalla)

Improvement

v3.8.3

  • Add support for setting test_case_id in update_current_trace, and include it when serializing traces. This makes it easier to associate a trace with a specific test case in downstream processing. (#2463) (Kritin Vongthongsri)

v3.8.2

  • Remove the New Relic OpenTelemetry tracing exporter from telemetry. This reduces external tracing overhead and avoids requiring New Relic-related tracing setup during event capture. (#2364) (Kritin Vongthongsri)
  • Improve LangChain and LangGraph integrations by stabilizing tracing and metric collection. Fix schema and integration test behavior to be deterministic, representative of supported usage, and aligned with the documentation. (#2457) (Trevor Wilson)

v3.8.1

  • Fix grammar in the README to improve clarity when describing locally run evaluation models and metrics. (#2423) (yuri)
  • Add a 2024 changelog page to the documentation and link it from the changelog index and sidebar for easier navigation. Update the changelog generator default output directory to match the new docs path. (#2429) (Trevor Wilson)
  • Improve contextual recall verdicts by attaching the expected_output to each verdict, making results easier to interpret and debug in both sync and async runs. (#2449) (Vamshi Adimalla)
  • Add support for passing a confident_api_key to dataset and prompt objects, and use it automatically for push/pull/update/queue operations. This makes it easier to work with multiple API keys without relying on a single global setting. (#2453) (Vamshi Adimalla)

v3.7.9

  • Improve environment variable docs by clarifying how boolean flags are parsed, including accepted truthy/falsy tokens and how unset or unrecognized values fall back to defaults. Update env var tables to show 1/0/unset for boolean settings. (#2399) (Trevor Wilson)
  • Fix a typo in the getting started docs to improve readability and clarity. (#2404) (Neelay Shah)
  • Update GEval documentation to reference evaluation_steps (instead of evaluation_params) when describing which parameters should be included for accurate results. (#2406) (Vishnu Sai Teja)
  • Fix Gemini defaults and documentation to avoid retired model IDs. Update the default model to gemini-2.5-pro and refresh Gemini/Vertex AI docs to use current stable/preview models and *-latest aliases. Update custom LLM guide examples to gemini-2.5-flash. (#2414) (Trevor Wilson)

v3.7.8

  • Improve OpenTelemetry export configuration by introducing CONFIDENT_OTEL_URL (defaulting to the hosted endpoint) and using it across integrations. This makes it easier to point tracing to regional endpoints such as the EU collector via an environment variable. (#2400) (Vamshi Adimalla)

Bug Fix

v3.8.3

  • Fix the CrewAI integration tests by improving event loop handling in sync wrappers and correctly comparing tool usage when traces are returned as lists. This prevents failures caused by missing loops and mismatched tools-used invariants. (#2460) (Vamshi Adimalla)

v3.8.2

  • Fix answer relevancy scoring by raising an error when actual_output is empty or whitespace-only, preventing blank outputs from being treated as fully relevant. (#2451) (Trevor Wilson)
  • Fix Confident API request routing by inferring the region from the API key prefix when no region is set. EU keys (confident_eu_...) now automatically use the EU endpoint, preventing invalid API key errors. Defaults to the US endpoint when the region cannot be inferred. (#2456) (Trevor Wilson)
  • Fix Pydantic AI OpenTelemetry instrumentation by setting the global tracer provider when possible and warning if it’s already configured. Improve agent span detection by reading agent names from gen_ai.agent.name or pydantic_ai.agent.name and applying agent attributes consistently at span start/end. (#2459) (Vamshi Adimalla)
  • Fix PydanticAI tracing environment selection so it prefers the trace manager setting, then CONFIDENT_TRACE_ENVIRONMENT, and defaults to development only when neither is set. (#2462) (Vamshi Adimalla)

v3.8.1

  • Fix LangChain/LangGraph callback tracing to reuse active traces, restore trace/span context across async tasks, and keep correct parent-child span hierarchy. Also avoid overwriting trace metadata when values are not provided. (#2434) (Jeffrey Ip)
  • Fix Amazon Bedrock Converse response parsing by extracting all text content blocks and ignoring reasoningContent. Improve error messages when no text is returned, and return None for cost when pricing data is unavailable. (#2437) (Trevor Wilson)
  • Fix incorrect return type annotations for send_annotation() and a_send_annotation() by changing them from str to None, matching their actual no-return behavior. (#2441) (yuri)
  • Fix multimodal image metrics to fail fast with a clear ValueError when actual_output contains no images. Also validate expected_output when detecting multimodal test cases and improve error messaging for mismatched output image counts. (#2447) (Vamshi Adimalla)

v3.7.9

  • Fix batch scoring on the DROP benchmark to use quasi_contains_score, matching single prediction behavior. This prevents partial matches like '2' being incorrectly marked wrong when the gold answer includes variants such as '2, 2-yards'. (#2402) (Aadam Haq)
  • Fix progress updates to ignore missing tasks during teardown, preventing StopIteration when async callbacks run after a progress task is removed. Progress updates now safely become a no-op in this race condition. (#2405) (Trevor Wilson)
  • Fix MMLU batch_predict to require batch_generate to return a list of MultipleChoiceSchema and raise a clear TypeError otherwise, preventing inconsistent response handling. (#2408) (Aadam Haq)
  • Fix Gemini Vertex AI authentication to fall back to Application Default Credentials when no service account key is provided, instead of requiring GOOGLE_SERVICE_ACCOUNT_KEY. Only parse and validate the key when present to avoid unnecessary OAuth imports. (#2412) (Trevor Wilson)
  • Fix Synthesizer export and save for conversational goldens: to_pandas and save_as now handle both QA and conversational outputs, include the right fields, and raise an error only when neither type is present. (#2415) (Vamshi Adimalla)

On this page