Vertex AI

You can also use Google Cloud's Vertex AI models, including Gemini or your own fine-tuned models, with DeepEval.

info

To use Vertex AI, you must have the following:

1. A Google Cloud project with the Vertex AI API enabled
2. Application Default Credentials set up:

gcloud auth application-default login

Command Line

Run the following command in your terminal to configure your deepeval environment to use Gemini models through Vertex AI for all metrics.

deepeval set-gemini \
    --model-name=<model_name> \ # e.g. "gemini-2.0-flash-001"
    --project-id=<project_id> \
    --location=<location> # e.g. "us-central1"

info

The CLI command above sets Gemini (via Vertex AI) as the default provider for all metrics, unless overridden in Python code. To use a different default model provider, you must first unset Gemini:

deepeval unset-gemini

Persisting settings

You can persist CLI settings with the optional --save flag. See Flags and Configs -> Persisting CLI settings.

Python

Alternatively, you can specify your model directly in code using GeminiModel from DeepEval's model collection. By default, model_name is set to gemini-1.5-pro.

from deepeval.models import GeminiModel
from deepeval.metrics import AnswerRelevancyMetric

model = GeminiModel(
    model_name="gemini-1.5-pro",
    project="Your Project ID",
    location="us-central1",
    temperature=0
)

answer_relevancy = AnswerRelevancyMetric(model=model)

There are THREE mandatory and ONE optional parameters when creating an GeminiModel through Vertex AI:

• model_name: A string specifying the name of the Gemini model to use.
• project: A string specifying your Google Cloud project ID.
• location: A string specifying the Google Cloud location.
• [Optional] temperature: A float specifying the model temperature. Defaulted to 0.
• [Optional] generation_kwargs: A dictionary of additional generation parameters supported by your model provider.

tip

Any **kwargs you would like to use for your model can be passed through the generation_kwargs parameter. However, we request you to double check the params supported by the model and your model provider in their official docs.

Available Vertex AI Models

note

This list only displays some of the available models. For a comprehensive list, refer to the Vertex AI's official documentation.

Below is a list of commonly used Gemini models:

gemini-2.0-pro-exp-02-05
gemini-2.0-flash
gemini-2.0-flash-001
gemini-2.0-flash-002
gemini-2.0-flash-lite
gemini-2.0-flash-lite-001
gemini-1.5-pro
gemini-1.5-pro-001
gemini-1.5-pro-002
gemini-1.5-flash
gemini-1.5-flash-001
gemini-1.5-flash-002
gemini-1.0-pro
gemini-1.0-pro-001
gemini-1.0-pro-002
gemini-1.0-pro-vision
gemini-1.0-pro-vision-001