Generate Goldens From Goldens

Q: Does it require context to work?

No. If your existing goldens include context, it grounds generation in that context; otherwise it falls back to generate_goldens_from_scratch using your existing inputs. Note expected_outputs are only generated when the seed goldens contain context.

deepeval enables you to generate synthetic goldens from an existing set of goldens, without requiring any documents or context. This is ideal for quickly expanding or adding more complexity to your evaluation dataset.

Existing goldens with input and expected output are reduced to contexts, which then generate new synthetic goldens of the same shape

Generate Your Goldens

To get started, simply define a Synthesizer object and pass in your list of existing goldens. Note that you can only generate single-turn goldens from existing single-turn ones, and vice versa.

from deepeval.synthesizer import Synthesizer

synthesizer = Synthesizer()
goldens = synthesizer.generate_goldens_from_goldens(
  goldens=goldens,
  max_goldens_per_golden=2,
  include_expected_output=True,
)

There is ONE mandatory and TWO optional parameter when using the generate_goldens_from_goldens method:

goldens: a list of existing Goldens from which the new Goldens will be generated.
[Optional] max_goldens_per_golden: the maximum number of goldens to be generated per golden. Defaulted to 2.
[Optional] include_expected_output: a boolean which when set to True, will additionally generate an expected_output for each synthetic Golden. Defaulted to True.

from deepeval.synthesizer import Synthesizer

synthesizer = Synthesizer()
conversational_goldens = synthesizer.generate_conversational_goldens_from_goldens(
  goldens=goldens,
  max_goldens_per_golden=2,
  include_expected_outcome=True,
)

There is ONE mandatory and TWO optional parameter when using the generate_conversational_goldens_from_goldens method:

goldens: a list of existing Goldens from which the new Goldens will be generated.
[Optional] max_goldens_per_golden: the maximum number of goldens to be generated per golden. Defaulted to 2.
[Optional] include_expected_outcome: a boolean which when set to True, will additionally generate an expected_outcome for each synthetic ConversationalGolden. Defaulted to True.

FAQs

When should I use generate_goldens_from_goldens?

Use it when you already have a set of seed goldens and want to quickly expand or add complexity to your dataset — no documents or contexts required. Control volume with max_goldens_per_golden.

Does it require context to work?

No. If your existing goldens include context, it grounds generation in that context; otherwise it falls back to generate_goldens_from_scratch using your existing inputs. Note expected_outputs are only generated when the seed goldens contain context.

Can I mix single-turn and multi-turn goldens?

No. You can only generate single-turn goldens from single-turn ones, and multi-turn ConversationalGoldens from multi-turn ones.

Where should the goldens I expand from come from?

Any goldens work, but the best seeds are cases your app has actually failed on. If you're already in production, Confident AI surfaces failing production traces and turns them into goldens, which you can then expand into variations with generate_goldens_from_goldens().

Generate Your Goldens

FAQs

On this page