Conversation Simulator

deepeval's ConversationSimulator allows you to simulate full conversations between a fake user and your chatbot, unlike the synthesizer which generates regular goldens representing single, atomic LLM interactions.

main.py
from deepeval.test_case import Turn
from deepeval.simulator import ConversationSimulator
from deepeval.dataset import ConversationalGolden

# Create ConversationalGolden
conversation_golden = ConversationalGolden(
    scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
    expected_outcome="Successful purchase of a ticket.",
    user_description="Andy Byron is the CEO of Astronomer.",
)

# Define chatbot callback
async def chatbot_callback(input):
    return Turn(role="assistant", content=f"Chatbot response to: {input}")

# Run Simulation
simulator = ConversationSimulator(model_callback=chatbot_callback)
conversational_test_cases = simulator.simulate(conversational_goldens=[conversation_golden])
print(conversational_test_cases)

The ConversationSimulator uses the scenario and user description from a ConversationalGolden to simulate back-and-forth exchanges with your chatbot. The resulting dialogue is used to create ConversationalTestCases for evaluation using deepeval's multi-turn metrics.

Create Your First Simulator

To create a ConversationSimulator, you'll need to define a callback that wraps around your LLM chatbot.

from deepeval.test_case import Turn
from deepeval.simulator import ConversationSimulator

async def model_callback(input: str, turns: List[Turn], thread_id: str) -> Turn:
    return Turn(role="assistant", content=f"I don't know how to answer this: {input}")

simulator = ConversationSimulator(model_callback=model_callback)

There are ONE mandatory and FOUR optional parameters when creating a ConversationSimulator:

model_callback: a callback that wraps around your conversational agent.
[Optional] simulator_model: a string specifying which of OpenAI's GPT models to use for generation, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-4.1.
[Optional] async_mode: a boolean which when set to True, enables concurrent simulation of conversations. Defaulted to True.
[Optional] max_concurrent: an integer that determines the maximum number of conversations that can be generated in parallel at any point in time. You can decrease this value if you're running into rate limit errors. Defaulted to 100.

Model callback

Only the input argument is required when defining your model_callback, but you may also define these optional arguments:

[Optional] turns: a list of Turns, which include the role and content of each message in the conversation.
[Optional] thread_id: a unique identifier for each conversation.

While turns captures the dialogue context for each turn, some applications must persist additional state across turns — for example, when invoking external APIs or tracking user-specific data. In these cases, you'll want to take advantage of the thread_id.

main.py
from deepeval.test_case import Turn

async def model_callback(input: str, turns: List[Turn], thread_id: str) -> Turn:

    # Inspect the turns and thread_id
    print(turns)
    print(thread_id)

    # Replace with your chatbot
    res = await your_llm_app(input, turns, thread_id)
    return Turn(role="assistant", content=res)

Simulate A Conversation

To simulate your first conversation, simply pass in a list of ConversationalGoldens to the simulate method:

from deepeval.dataset import ConversationalGolden
...

conversation_golden = ConversationalGolden(
    scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
    expected_outcome="Successful purchase of a ticket.",
    user_description="Andy Byron is the CEO of Astronomer.",
)
conversational_test_cases = simulator.simulate(conversational_goldens=[conversation_golden])

There are ONE mandatory and ONE optional parameter when calling the simulate method:

conversational_goldens: a list of ConversationalGoldens that specify the scenario and user description.
[Optional] max_user_simulations: an integer that specifies the maximum number of user-assistant message cycles to simulate per conversation. Defaulted to 10.

A simulation ends either when the converaiton achieves the expected outcome outlined in a ConversationalGolden, or when the max_user_simulations has been reached.

tip

You can also generate conversations from existing turns. Simply populate your ConversationalGolden with a list of initial Turns, and the simulator will continue the conversation.

Advanced Usage

Lifecycle hooks

The ConversationSimulator provides a on_simulation_complete hooks that allow you to execute custom logic whenever a simulation of an individual test case has completed. This allows you to process each ConversationalTestCase as soon as it's generated, rather than waiting for all simulations to finish.

from deepeval.simulator import ConversationSimulator
from deepeval.test_case import ConversationalTestCase

def handle_simulation_complete(test_case: ConversationalTestCase, index: int):
    print(f"Conversation {index} completed with {len(test_case.turns)} turns")

conversational_test_cases = simulator.simulate(
    conversational_goldens=[golden1, golden2, golden3],
    on_simulation_complete=handle_simulation_complete
)

The hook function receives two parameters:

test_case: the completed ConversationalTestCase object containing all turns and metadata.
index: the index of the corresponding golden that was simulated (ording is perserved during simulation).

tip

When using async_mode=True, conversations may complete in any order due to concurrent execution. Use the index parameter to track which golden each test case corresponds to.

Existing turns

If your multi-turn chatbot has one or more predefined turns (for example, a hardcoded assistant message at the beginning of a conversation), you would simply include this as part of the simulation by providing a list of preexisting turns to a ConversationalGolden:

from deepeval.test_case import ConversationalTestCase, Turn

golden = ConversationalGolden(turns=[Turn(role="assistant", content="Hi! How can I help you today?")])

By including a list of non-empty turns, deepeval will run simulations based on the additional context you've provided.

Evaluate Simulated Turns

The simulate function returns a list of ConversationalTestCases, which can be used to evaluate your LLM chatbot using deepeval's conversational metrics. Use simulated conversations to run end-to-end evaluations:

from deepeval import evaluate
from deepeval.metrics import TurnRelevancyMetric
...

evaluate(test_cases=conversational_test_cases, metrics=[TurnRelevancyMetric()])

Create Your First Simulator​

Model callback​

Simulate A Conversation​

Advanced Usage​

Lifecycle hooks​

Existing turns​

Evaluate Simulated Turns​