Json Correctness

Q: Why do I still need to pass a model if the metric is deterministic?

Scoring is deterministic — it only checks whether actual_output loads into your expected_schema. The model only generates the reason on invalid schemas when include_reason is True; set include_reason=False to skip it.

Q: How do I validate an output that's a list of JSON objects?

Wrap your pydantic BaseModel in a RootModel — e.g. class ExampleSchemaList(RootModel[List[ExampleSchema]]) — and pass it as the expected_schema.

Q: When should I use Json Correctness over an LLM-judge metric?

When you care about structure rather than meaning — verifying tool calls, function arguments, or API responses parse into the expected fields and types. To judge the JSON's contents, pair it with an LLM-based metric.

Single-turn

Referenceless

The json correctness metric measures whether your LLM application is able to generate actual_outputs with the correct json schema.

Required Arguments

To use the JsonCorrectnessMetric, you'll have to provide the following arguments when creating an LLMTestCase:

input
actual_output

Read the How Is It Calculated section below to learn how test case parameters are used for metric calculation.

Usage

First define your schema by creating a pydantic BaseModel:

from pydantic import BaseModel

class ExampleSchema(BaseModel):
    name: str

If your actual_output is a list of JSON objects, you can simply create a list schema by wrapping your existing schema in a RootModel. For example:

from pydantic import RootModel
from typing import List

...

class ExampleSchemaList(RootModel[List[ExampleSchema]]):
    pass

Then supply it as the expected_schema when creating a JsonCorrectnessMetric, which can be used for end-to-end evaluation:

from deepeval import evaluate
from deepeval.metrics import JsonCorrectnessMetric
from deepeval.test_case import LLMTestCase


metric = JsonCorrectnessMetric(
    expected_schema=ExampleSchema,
    model="gpt-4",
    include_reason=True
)
test_case = LLMTestCase(
    input="Output me a random Json with the 'name' key",
    # Replace this with the actual output from your LLM application
    actual_output="{'name': null}"
)

# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[test_case], metrics=[metric])

There are ONE mandatory and SEVEN optional parameters when creating an PromptAlignmentMetric:

expected_schema: a pydantic BaseModel specifying the schema of the Json that is expected from your LLM.
[Optional] threshold: a float representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] model: a string specifying which of OpenAI's GPT models to use to generate reasons, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
[Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
[Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
[Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
[Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
[Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.

Within components

You can also run the JsonCorrectnessMetric within nested components for component-level evaluation.

from deepeval.dataset import Golden
from deepeval.tracing import observe, update_current_span
...

@observe(metrics=[metric])
def inner_component():
    # Set test case at runtime
    test_case = LLMTestCase(input="...", actual_output="...")
    update_current_span(test_case=test_case)
    return

@observe
def llm_app(input: str):
    # Component can be anything from an LLM call, retrieval, agent, tool use, etc.
    inner_component()
    return

evaluate(observed_callback=llm_app, goldens=[Golden(input="Hi!")])

As a standalone

You can also run the JsonCorrectnessMetric on a single test case as a standalone, one-off execution.

...

metric.measure(test_case)
print(metric.score, metric.reason)

How Is It Calculated?

The PromptAlignmentMetric score is calculated according to the following equation:

\text{Json Correctness} = \begin{cases} 1 & \text{If the actual output fits the expected schema}, \\ 0 & \text{Otherwise} \end{cases}

The JsonCorrectnessMetric does not use an LLM for evaluation and instead uses the provided expected_schema to determine whether the actual_output can be loaded into the schema.

FAQs

Why do I still need to pass a model if the metric is deterministic?

Scoring is deterministic — it only checks whether actual_output loads into your expected_schema. The model only generates the reason on invalid schemas when include_reason is True; set include_reason=False to skip it.

Does scoring cost money or call an LLM?

Schema validation is free and deterministic. A model is only invoked — incurring token cost — when an output fails the schema and include_reason is True.

How do I validate an output that's a list of JSON objects?

Wrap your pydantic BaseModel in a RootModel — e.g. class ExampleSchemaList(RootModel[List[ExampleSchema]]) — and pass it as the expected_schema.

When should I use Json Correctness over an LLM-judge metric?

When you care about structure rather than meaning — verifying tool calls, function arguments, or API responses parse into the expected fields and types. To judge the JSON's contents, pair it with an LLM-based metric.

Required Arguments

Usage

Within components

As a standalone

How Is It Calculated?

FAQs

On this page