vLLM

vLLM is a high-performance inference engine for LLMs that supports OpenAI-compatible APIs. deepeval can connect to a running vLLM server for running local evaluations.

Command Line

1. Launch your vLLM server and ensure it’s exposing the OpenAI-compatible API. The typical base URL for a local vLLM server is: http://localhost:8000/v1/.
2. Then run the following command to configure deepeval:

deepeval set-local-model \
    --model=<model_name> \
    --base-url="http://localhost:8000/v1/"

Reverting to OpenAI

To disable the local model and return to OpenAI:

deepeval unset-local-model