CLI Settings

Quick Summary

deepeval provides a CLI for managing common tasks directly from the terminal. You can use it for:

• Logging in/out and viewing test runs
• Running evaluations from test files
• Checking a project against its governance policy (deepeval gate)
• Diagnosing your environment (deepeval diagnose): default models, effective settings, and where each value comes from
• Enabling/disabling debug
• Selecting an LLM provider (OpenAI, Azure OpenAI, Gemini, Grok, DeepSeek, OpenRouter, local/Ollama)
• Setting/unsetting provider-specific options (model, endpoint, deployment, etc.)
• Listing and updating any deepeval setting (deepeval settings -l, deepeval settings --set KEY=VALUE)
• Saving settings and secrets persistently to .env files
• Inspecting saved test runs in a terminal TUI (deepeval inspect)

• Generating synthetic goldens from docs, contexts, scratch, or existing goldens
• Selecting an embeddings provider (Azure OpenAI, local, Ollama)

Install & Update

pip install -U deepeval

To review available commands consult the CLI built in help:

deepeval --help

Read & Write Settings

deepeval reads settings from dotenv files in the current working directory, without overriding existing process environment variables. Dotenv precedence (lowest → highest) is: .env → .env.<APP_ENV> → .env.local.

deepeval also uses a legacy JSON keystore at .deepeval/.deepeval for non-secret keys. This keystore is treated as a fallback (dotenv/process env take precedence). Secrets are never written to the JSON keystore.

Core Commands

generate

Use deepeval generate to generate synthetic goldens from the terminal with the Golden Synthesizer. The command requires two selectors:

• --method: where goldens come from: docs, contexts, scratch, or goldens
• --variation: what to generate: single-turn or multi-turn

Generate single-turn goldens from documents:

deepeval generate \
  --method docs \
  --variation single-turn \
  --documents example.txt \
  --documents another.pdf \
  --output-dir ./synthetic_data

Generate multi-turn goldens from scratch:

deepeval generate \
  --method scratch \
  --variation multi-turn \
  --num-goldens 25 \
  --scenario-context "Users asking support questions" \
  --conversational-task "Help users solve product issues" \
  --participant-roles "User and assistant"

Common options:

Method-specific options:

For a deeper walkthrough, see the Golden Synthesizer docs.

test

Use deepeval test run to run evaluation test files through pytest with the deepeval pytest plugin enabled.

deepeval test --help
deepeval test run --help

Run a single test file:

deepeval test run test_chatbot.py

Run a test directory:

deepeval test run tests/evals

Run a specific test:

deepeval test run test_chatbot.py::test_answer_relevancy

Useful options:

You can pass additional pytest flags after the deepeval options. For example:

deepeval test run tests/evals \
  --mark "not slow" \
  --exit-on-first-failure \
  -- --tb=short

inspect

Use deepeval inspect to open a saved test run inside a terminal TUI — a trace-tree viewer for metric scores, reasons, inputs/outputs, tool calls, and retriever context, all without leaving the terminal.

The TUI is a trace viewer, so it's only useful for runs whose test cases captured a trace from an instrumented app. Every such run writes a rolling snapshot into the gitignored .deepeval cache dir, so the zero-arg form picks up the most recent one automatically:

Traces come from evals_iterator() against an agent instrumented with @observe (see single-turn end-to-end evals). Each call rewrites .deepeval/.latest_run_full.json.

deepeval inspect

You can also point it at a specific file or folder:

deepeval inspect ./experiments/test_run_20260512_174200.json
deepeval inspect ./experiments          # latest test_run_*.json inside
deepeval inspect --folder ./experiments # same, via explicit flag

This allows you to inspect traces and spans locally on your machine:

Move with ↑/↓ or j/k, fold a span with h/l, and cycle between traces with ←/→ or n/p. Press / to filter spans by name, y to copy the selected node as JSON (Y for the whole trace), ? for the keybinding list, and q to quit.

Resolution order when no path is passed: --folder → DEEPEVAL_RESULTS_FOLDER → .deepeval/.latest_run_full.json → ./experiments (legacy fallback).

The TUI needs an optional extras bundle (Textual + clipboard support):

pip install 'deepeval[inspect]'

diagnose

Use deepeval diagnose to print the configuration your environment actually resolves to — and where each value comes from. Reach for it when evals are using the wrong model, credentials, or data region, or when you're not sure which .env file is winning.

deepeval diagnose

The report shows:

• Versions: deepeval and the runtime it's running on.
• Evaluation model: the model deepeval will use when a metric (or synthesizer, simulator, etc.) is constructed without an explicit model, plus the reason it won. The default embedding model is listed alongside it, with what each is used by.
• Configuration sources: the precedence order every variable is resolved against, independently (process environment → .env.local → .env.<APP_ENV> → .env → JSON keystore → built-in defaults), which dotenv files were loaded, and where the keystore lives.
• Configured settings: every deepeval setting that is explicitly set, with secrets masked (only the last 6 characters shown) and the winning source for each — the report uses the same resolvers as the runtime, so what you see is what your evals get.
• Confident AI: login status, masked API key, data region, and the API endpoint. The OTEL endpoint is shown too, with warnings if either endpoint looks like it belongs to a different region than your API key.

Options:

Confident AI Commands

Use these commands to connect deepeval to Confident AI (deepeval Cloud) so your local evaluations can be uploaded, organized, and viewed as rich test run reports on the cloud. If you don’t have an account yet, sign up here.

login & logout

• deepeval login [--api-key ...] [--save=dotenv[:path]] : Log in to Confident AI. The interactive flow opens a browser for authentication, then returns to the terminal to create an organization and first project or select an existing project. DeepEval creates a dedicated project API key and saves it as CONFIDENT_API_KEY. Use --api-key for CI, headless automation, or manual fallback.
• deepeval logout [--save=dotenv[:path]]: Remove your Confident AI credentials from local persistence — the JSON keystore and every dotenv file deepeval auto-loads (.env, .env.<APP_ENV>, .env.local). If CONFIDENT_API_KEY is exported by your shell itself, deepeval cannot unset it and will tell you to run unset CONFIDENT_API_KEY.

view

deepeval view opens the latest test run on Confident AI in your browser. If needed, it uploads the cached run artifacts first. It takes no flags.

set-confident-region

Set the data region your project's data lives in, which also determines the Confident AI endpoint deepeval talks to. See data residency for what each region means.

deepeval set-confident-region EU --save=dotenv

gate

Check your project against its governance policy and exit with a non-zero code when it doesn't pass, which makes it usable as a CI gate. See policies for how to define one.

deepeval gate

Persistence & Secrets

All set-* / unset-* commands follow the same rules:

• Non-secrets (model name, endpoint, deployment, etc.) may be mirrored into .deepeval/.deepeval.
• Secrets (API keys) are never written to .deepeval/.deepeval.
• Pass --save=dotenv[:path] to write settings (including secrets) to a dotenv file (default: .env.local).
• If --save is omitted, deepeval will use DEEPEVAL_DEFAULT_SAVE if set; otherwise it won’t write a dotenv file (some commands like login still default to .env.local).
• Unsetting one provider only removes that provider’s keys. Leftover credentials from other providers (e.g. OPENAI_API_KEY) don’t select anything on their own — the USE_* flags decide which provider is active.

To set the model and token cost for Anthropic, and then view the settings it wrote, you would run:

deepeval set-anthropic -m claude-3-7-sonnet-latest -i 0.000003 -o 0.000015 --save=dotenv
Saved environment variables to .env.local (ensure it's git-ignored).
🙌 Congratulations! You're now using Anthropic `claude-3-7-sonnet-latest` for all evals that require an LLM.

deepeval settings -l anthropic
                                                                                Settings
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                            ┃ Value                    ┃ Description                                                                                      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ANTHROPIC_API_KEY               │ ********                 │ Anthropic API key.                                                                               │
│ ANTHROPIC_COST_PER_INPUT_TOKEN  │ 3e-06                    │ Anthropic input token cost (used for cost reporting).                                            │
│ ANTHROPIC_COST_PER_OUTPUT_TOKEN │ 1.5e-05                  │ Anthropic output token cost (used for cost reporting).                                           │
│ ANTHROPIC_MODEL_NAME            │ claude-3-7-sonnet-latest │ Anthropic model name (e.g. 'claude-3-...').                                                      │
│ USE_ANTHROPIC_MODEL             │ True                     │ Select Anthropic as the active LLM provider (USE_* flags are mutually exclusive in CLI helpers). │
└─────────────────────────────────┴──────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘

Debug Controls

Use these to turn on structured logs, gRPC wire tracing, and Confident tracing (all optional).

deepeval set-debug \
  --log-level DEBUG \
  --debug-async \
  --retry-before-level INFO \
  --retry-after-level ERROR \
  --grpc --grpc-verbosity DEBUG --grpc-trace list_tracers \
  --trace-verbose --trace-env staging --trace-flush \
  --save=dotenv

• Immediate effect in the current process
• Optional persistence via --save=dotenv[:path]
• No-op guard: If nothing would change, you’ll see No changes to save … (and nothing is written).

To filter (substring match) settings by name, displaying each setting's current value and description, pass the filter to settings -l:

deepeval settings -l log-level
                                                            Settings
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                            ┃ Value ┃ Description                                                                  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ DEEPEVAL_RETRY_AFTER_LOG_LEVEL  │ 20    │ Log level for 'after retry' logs (defaults to ERROR).                        │
│ DEEPEVAL_RETRY_BEFORE_LOG_LEVEL │ 20    │ Log level for 'before retry' logs (defaults to LOG_LEVEL if set, else INFO). │
│ LOG_LEVEL                       │ 40    │ Global logging level (e.g. DEBUG/INFO/WARNING/ERROR/CRITICAL or numeric).    │
└─────────────────────────────────┴───────┴──────────────────────────────────────────────────────────────────────────────┘

To restore defaults and clean persisted values:

deepeval unset-debug --save=dotenv

Model Provider Configs

All provider commands come in pairs:

• deepeval set-<provider> [provider-specific flags] [--save=dotenv[:path]] [--quiet]
• deepeval unset-<provider> [--save=dotenv[:path]] [--quiet]

This switches the active provider:

• It sets USE_<PROVIDER>_MODEL = True for the chosen provider, and
• Turns all other USE_* flags off so that only one provider is enabled at a time.

When you set a provider, the CLI enables that provider’s USE_<PROVIDER>_MODEL flag and disables all other USE_* flags. When you unset a provider, it disables only that provider’s USE_* flag and leaves all others untouched. If you manually set env vars (or edit dotenv files) it’s possible to end up with multiple USE_* flags enabled.

Full model list

Embeddings:

Common Issues

• Nothing printed? For set-* / unset-* / set-debug, a clean exit with no output often means you are passing the --quiet / -q flag.
• Provider still active after unsetting? Unsetting turns off that provider's USE_* flag only; if another provider's flag is still on, it becomes the active provider. With no USE_* flag enabled, deepeval falls back to OpenAI. To force a provider, run the corresponding set-<provider> command.
• Dotenv edits not picked up? deepeval loads dotenv files from the current working directory by default, or ENV_DIR_PATH if set. Ensure your process runs in that context.

If you’re still stuck, the dedicated Troubleshooting page covers deeper debugging (TLS errors, logging, timeouts, dotenv loading, and config caching).