Flags and Configs

Sometimes you might want to customize the behavior of different settings for evaluate() and assert_test(), and this can be done using "configs" (short for configurations) and "flags".

Configs for evaluate()

Each config is a dataclass imported from deepeval.evaluate, and each is passed to evaluate() under its own keyword argument.

Async Configs

The AsyncConfig controls how concurrently work is dispatched during evaluate().

from deepeval.evaluate import AsyncConfig
from deepeval import evaluate

evaluate(async_config=AsyncConfig(), ...)

There are THREE optional parameters when creating an AsyncConfig:

• [Optional] run_async: a boolean which when set to True, enables concurrent evaluation of test cases AND metrics. Defaulted to True.
• [Optional] throttle_value: an integer that determines how long (in seconds) to throttle the evaluation of each test case. You can increase this value if your evaluation model is running into rate limit errors. Defaulted to 0.
• [Optional] max_concurrent: an integer that determines the maximum number of test cases that can be ran in parallel at any point in time. You can decrease this value if your evaluation model is running into rate limit errors. Defaulted to 20.

The throttle_value and max_concurrent parameter is only used when run_async is set to True. A combination of a throttle_value and max_concurrent is the best way to handle rate limiting errors, either in your LLM judge or LLM application, when running evaluations.

Display Configs

The DisplayConfig controls how results and intermediate execution steps are displayed during evaluate().

from deepeval.evaluate import DisplayConfig
from deepeval import evaluate

evaluate(display_config=DisplayConfig(), ...)

There are TEN optional parameters when creating a DisplayConfig:

• [Optional] verbose_mode: a optional boolean which when IS NOT None, overrides each metric's verbose_mode value. Defaulted to None.
• [Optional] display: a str of either "all", "failing" or "passing", which allows you to selectively decide which type of test cases to display as the final result. Defaulted to "all".
• [Optional] show_indicator: a boolean which when set to True, shows the evaluation progress indicator for each individual metric. Defaulted to True.
• [Optional] print_results: a boolean which when set to True, prints the result of each evaluation. Defaulted to True.
• [Optional] results_folder: a string path to a directory where each call to evaluate() (or evals_iterator()) will be persisted as a test_run_<YYYYMMDD_HHMMSS>.json file. Defaulted to None (no local save). See Saving test runs locally below.
• [Optional] results_subfolder: an optional string that, when set together with results_folder, nests the test_run_*.json files under results_folder/results_subfolder/. Defaulted to None (flat layout).
• [Optional] truncate_passing_cases: a boolean which when set to True, truncates the terminal output of passing test cases. Defaulted to True.
• [Optional] inspect_after_run: a boolean which when set to True, prompts you at the end of an evals_iterator() run to open the captured traces in the deepeval inspect TUI. Only fires in interactive terminals when at least one test case has a trace. Set to False to disable per call, or export DEEPEVAL_NO_INSPECT_PROMPT=1 to disable globally (e.g. in CI). Defaulted to True.
• [Optional] file_type: a string of either "html" or "md", which allows you to export the evaluation dashboard to a file. Defaulted to None.
• [Optional] file_output_dir: a string which when set, writes the evaluation dashboard to the specified directory using the format specified in file_type. Defaulted to None.

Saving test runs locally

Runs can be persisted to disk as a structured TestRun JSON. Hyperparameters, per-test-case scores, and metric reasons are all serialized into each file via the same schema that Confident AI uses — no extra setup required.

Set results_folder to persist every evaluate() (or evals_iterator()) call:

from deepeval import evaluate
from deepeval.evaluate import DisplayConfig

for temp in [0.0, 0.4, 0.8]:
    evaluate(
        test_cases=test_cases,
        metrics=metrics,
        hyperparameters={"model": "gpt-4o-mini", "temperature": temp},
        display_config=DisplayConfig(results_folder="./evals/prompt-v3"),
    )

After a few runs, the folder is flat — just the raw test runs:

./evals/prompt-v3/
  test_run_20260421_140114.json
  test_run_20260421_140132.json
  test_run_20260421_140151.json

The timestamp prefix makes ls order match chronological order, so an AI agent (Cursor, Claude Code) can iterate over the folder in the order runs happened.

If two runs finish within the same second, the writer appends _2, _3, … to the filename so nothing is ever overwritten.

Set results_subfolder to nest the runs under an extra directory — useful when the parent folder already holds other artifacts:

DisplayConfig(results_folder="./evals/prompt-v3", results_subfolder="test_runs")

./evals/prompt-v3/
  test_runs/
    test_run_20260421_140114.json
    test_run_20260421_140132.json

If results_folder is unset but the DEEPEVAL_RESULTS_FOLDER environment variable is present, deepeval falls back to that path for backwards compatibility.

Error Configs

The ErrorConfig controls how errors are handled in evaluate().

from deepeval.evaluate import ErrorConfig
from deepeval import evaluate

evaluate(error_config=ErrorConfig(), ...)

There are TWO optional parameters:

• [Optional] skip_on_missing_params: a boolean which when enabled, skips all metric executions for test cases with missing parameters. Defaulted to False.
• [Optional] ignore_errors: a boolean which when enabled, ignores all exceptions raised during metrics execution for each test case. Defaulted to False.

If both are enabled, skip_on_missing_params takes precedence. This means that if a metric is missing required test case parameters, it will be skipped (and the result will be missing) rather than appearing as an ignored error in the final test run.

Cache Configs

The CacheConfig controls the caching behavior of evaluate().

from deepeval.evaluate import CacheConfig
from deepeval import evaluate

evaluate(cache_config=CacheConfig(), ...)

There are TWO optional parameters:

• [Optional] use_cache: a boolean which when enabled, uses cached test run results instead. Defaulted to False.
• [Optional] write_cache: a boolean which when enabled, writes test run results to DISK. Defaulted to True.

Results are keyed by test case content plus metric configuration, so a cached result is only reused when both are unchanged. The write_cache parameter writes to disk and so you should disable it if that is causing any errors in your environment.

Hyperparameters

Log the model, prompt, and other configuration values with each test run so you can compare runs side-by-side on Confident AI and identify the best combination.

Pass them to evaluate() directly, as a dict of str | int | float or Prompt values:

from deepeval import evaluate

evaluate(
    test_cases=test_cases,
    metrics=metrics,
    hyperparameters={"model": "gpt-4.1", "temperature": 0.7, "prompt": prompt},
)

Under deepeval test run there is no evaluate() call to pass them to, so use the @deepeval.log_hyperparameters decorator anywhere in your test file instead:

import deepeval

@deepeval.log_hyperparameters
def hyperparameters():
    return {"model": "gpt-4.1", "temperature": 0.7}

Non-string values are stringified, and any key with an empty value is dropped. On Confident AI the logged values become filterable axes for comparing test runs and surfacing the configuration that performs best.

Flags for deepeval test run

Parallelization

Evaluate each test case in parallel by providing a number to the -n flag to specify how many processes to use.

deepeval test run test_example.py -n 4

Cache

Provide the -c flag (with no arguments) to read from the local deepeval cache instead of re-evaluating test cases on the same metrics.

deepeval test run test_example.py -c

Ignore Errors

Ignore errors for metric executions during a test run. An example of where this is helpful is if you're using a custom LLM and often find it generating invalid JSONs that will stop the execution of the entire test run.

The -i flag (with no arguments):

deepeval test run test_example.py -i

Verbose Mode

The -v flag (with no arguments) allows you to turn on verboseMode for all metrics. Not supplying the -v flag will default each metric's verboseMode to its value at instantiation.

deepeval test run test_example.py -v

Skip Test Cases

The -s flag (with no arguments) allows you to skip metric executions where the test case has missing/insufficient parameters (such as retrieval_context) that is required for evaluation. An example of where this is helpful is if you're using a metric such as the ContextualPrecisionMetric but don't want to apply it when the retrieval context is empty.

deepeval test run test_example.py -s

Identifier

Name test runs to better identify them on Confident AI. An example of where this is helpful is if you're running automated deployment pipelines, have deployment IDs, or just want a way to identify which test run is which for comparison purposes.

The -id flag, followed by a string:

deepeval test run test_example.py -id "My Latest Test Run"

When evaluating with evaluate() instead, pass identifier to achieve the same:

from deepeval import evaluate

evaluate(test_cases=[...], metrics=[...], identifier="My Latest Test Run")

Display Mode

The -d flag followed by a string of "all", "passing", or "failing" allows you to display only certain test cases in the terminal. For example, you can display "failing" only if you only care about the failing test cases.

deepeval test run test_example.py -d "failing"

Repeats

Repeat each test case by providing a number to the -r flag to specify how many times to rerun each test case.

deepeval test run test_example.py -r 2

Official Test Runs

The --official flag (or -o) marks the resulting test run as the official test run on Confident AI. The official run is used as the point of comparison for regression testing. This requires a CONFIDENT_API_KEY.

deepeval test run test_example.py --official

When evaluating with evaluate() instead, pass official=True to achieve the same:

from deepeval import evaluate

evaluate(test_cases=[...], metrics=[...], official=True)

Hooks

deepeval's Pytest integration allows you to run custom code at the end of each evaluation via the @deepeval.on_test_run_end decorator:

...

@deepeval.on_test_run_end
def function_to_be_called_after_test_run():
    print("Test finished!")

FAQs