Instruction Completeness Classifier

The InstructionCompletenessClassifier in deepeval is a categorical LLM-as-a-judge that determines whether your LLM application addressed every part of a multi-part request, only some parts, or none.

It is most useful with requests that have several explicit parts, since that is where an assistant tends to answer the first one and forget the rest.

Labels

• complete: every part of the user's request is addressed in the response.
• partial: some parts of the user's request are addressed and others are not.
• ignored: none of the parts of the user's request are addressed.

Usage

from deepeval.classifiers import InstructionCompletenessClassifier
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

classifier = InstructionCompletenessClassifier()

test_case = LLMTestCase(
    input="Summarize the meeting notes, list the action items, and suggest a date for the follow-up.",
    actual_output="Summary: ... Action items: 1. ... 2. ... Suggested follow-up: next Tuesday.",
    expected_labels={classifier.name: "complete"},
)

evaluate(test_cases=[test_case], classifiers=[classifier])

There are FIVE optional parameters when creating a InstructionCompletenessClassifier:

• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
• [Optional] include_reason: a boolean which when set to True, includes a reason for the chosen label. Defaulted to True.
• [Optional] allow_none: a boolean which when set to True, lets the classifier return no label when none of them fit (surfaced as label=None, with a reason). When False, the closest label is always chosen. Defaulted to False.
• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the classify() method. Defaulted to True.
• [Optional] classification_template: a subclass of ClassifierTemplate used to override the default prompts. Defaulted to ClassifierTemplate.

As a standalone

You can also run the InstructionCompletenessClassifier on a single test case as a standalone, one-off execution. classify() returns the label and stores the result on the instance:

...

label = classifier.classify(test_case)
print(classifier.label, classifier.reason)

How Is It Calculated?

The InstructionCompletenessClassifier is a one-shot LLM-as-a-judge: a single call to your evaluation model with the test case content and the labels above, returning the chosen label and a reason. The result is compared against the test case's expected_labels entry, if any, to decide pass or fail.