Skip to main content

Configure LLM

As described in the introduction, the default LLM and embedding is OpenAI due to its superior performance and speed. With that said, GPT Researcher supports various open/closed source LLMs and embeddings, and you can easily switch between them by updating the SMART_LLM, FAST_LLM and EMBEDDING env variables. You might also need to include the provider API key and corresponding configuration params.

Current supported LLMs are openai, anthropic, azure_openai, cohere, google_vertexai, google_genai, fireworks, ollama, together, mistralai, huggingface, groq and bedrock.

Current supported embeddings are openai, azure_openai, cohere, google_vertexai, google_genai, fireworks, ollama, together, mistralai, huggingface, nomic and voyageai.

To learn more about support customization options see here.

Please note: GPT Researcher is optimized and heavily tested on GPT models. Some other models might run into context limit errors, and unexpected responses. Please provide any feedback in our Discord community channel, so we can better improve the experience and performance.

Below you can find examples for how to configure the various supported LLMs.

OpenAI

# set the custom OpenAI API key
OPENAI_API_KEY=[Your Key]

# specify llms
FAST_LLM="openai:gpt-4o-mini"
SMART_LLM="openai:gpt-4o"
STRATEGIC_LLM="openai:o1-preview"

# specify embedding
EMBEDDING="openai:text-embedding-3-small"

Custom LLM

Create a local OpenAI API using llama.cpp Server.

For custom LLM, specify "openai:{your-llm}"

# set the custom OpenAI API url
OPENAI_BASE_URL="http://localhost:1234/v1"
# set the custom OpenAI API key
OPENAI_API_KEY="dummy_key"

# specify custom llms  
FAST_LLM="openai:your_fast_llm"
SMART_LLM="openai:your_smart_llm"
STRATEGIC_LLM="openai:your_strategic_llm"

For custom embedding, set "custom:{your-embedding}"

# set the custom OpenAI API url
OPENAI_BASE_URL="http://localhost:1234/v1"
# set the custom OpenAI API key
OPENAI_API_KEY="dummy_key"

# specify the custom embedding model   
EMBEDDING="custom:your_embedding"

Azure OpenAI

See also the documentation in the Langchain Azure OpenAI page.

On Azure OpenAI you will need to create deployments for each model you want to use. Please also specify the model names/deployment names in your .env file:

AZURE_OPENAI_API_KEY=[Your Key]
AZURE_OPENAI_ENDPOINT=https://{your-endpoint}.openai.azure.com/
OPENAI_API_VERSION=2024-05-01-preview

# note that the deployment name must be the same as the model name
FAST_LLM=azure_openai:gpt-4o-mini 
SMART_LLM=azure_openai:gpt-4o
STRATEGIC_LLM=azure_openai:o1-preview

EMBEDDING="azure_openai:text-embedding-ada-002"

Ollama

GPT Researcher supports both Ollama LLMs and embeddings. You can choose each or both. To use Ollama you can set the following environment variables

OLLAMA_BASE_URL=http://localhost:11434
FAST_LLM="ollama:llama3"
SMART_LLM="ollama:llama3"
STRATEGIC_LLM="ollama:llama3"

EMBEDDING="ollama:nomic-embed-text"

Groq

GroqCloud provides advanced AI hardware and software solutions designed to deliver amazingly fast AI inference performance. To leverage Groq in GPT-Researcher, you will need a GroqCloud account and an API Key. (NOTE: Groq has a very generous free tier.)

Sign up

Update env vars

And finally, you will need to configure the GPT-Researcher Provider and Model variables:

GROQ_API_KEY=[Your Key]

# Set one of the LLM models supported by Groq
FAST_LLM="groq:Mixtral-8x7b-32768"
SMART_LLM="groq:Mixtral-8x7b-32768"
STRATEGIC_LLM="groq:Mixtral-8x7b-32768"

NOTE: As of the writing of this Doc (May 2024), the available Language Models from Groq are:

  • Llama3-70b-8192
  • Llama3-8b-8192
  • Mixtral-8x7b-32768
  • Gemma-7b-it

Anthropic

Refer to Anthropic Getting started page to obtain Anthropic API key. Update the corresponding env vars, for example:

ANTHROPIC_API_KEY=[Your key]
FAST_LLM="anthropic:claude-2.1"
SMART_LLM="anthropic:claude-3-opus-20240229"
STRATEGIC_LLM="anthropic:claude-3-opus-20240229"

Anthropic does not offer its own embedding model.

Mistral AI

Sign up for a Mistral API key. Then update the corresponding env vars, for example:

MISTRAL_API_KEY=[Your key]
FAST_LLM="mistralai:open-mistral-7b"
SMART_LLM="mistralai:mistral-large-latest"
STRATEGIC_LLM="mistralai:mistral-large-latest"

EMBEDDING="mistralai:mistral-embed"

Together AI

Together AI offers an API to query 50+ leading open-source models in a couple lines of code. Then update corresponding env vars, for example:

TOGETHER_API_KEY=[Your key]
FAST_LLM="together:meta-llama/Llama-3-8b-chat-hf"
SMART_LLM="together:meta-llama/Llama-3-70b-chat-hf"
STRATEGIC_LLM="together:meta-llama/Llama-3-70b-chat-hf"

EMBEDDING="mistralai:nomic-ai/nomic-embed-text-v1.5"

HuggingFace

This integration requires a bit of extra work. Follow this guide to learn more. After you've followed the tutorial above, update the env vars:

HUGGINGFACE_API_KEY=[Your key]
FAST_LLM="huggingface:HuggingFaceH4/zephyr-7b-beta"
SMART_LLM="huggingface:HuggingFaceH4/zephyr-7b-beta"
STRATEGIC_LLM="huggingface:HuggingFaceH4/zephyr-7b-beta"

EMBEDDING="sentence-transformers/all-MiniLM-L6-v2"

Google Gemini

Sign up here for obtaining a Google Gemini API Key and update the following env vars:

GOOGLE_API_KEY=[Your key]
FAST_LLM="google_genai:gemini-1.5-flash"
SMART_LLM="google_genai:gemini-1.5-pro"
STRATEGIC_LLM="google_genai:gemini-1.5-pro"

EMBEDDING="google_genai:models/text-embedding-004"

Google VertexAI

FAST_LLM="google_vertexai:gemini-1.5-flash-001"
SMART_LLM="google_vertexai:gemini-1.5-pro-001"
STRATEGIC_LLM="google_vertexai:gemini-1.5-pro-001"

EMBEDDING="google_vertexai:text-embedding-004"

Cohere

COHERE_API_KEY=[Your key]
FAST_LLM="cohere:command"
SMART_LLM="cohere:command-nightly"
STRATEGIC_LLM="cohere:command-nightly"

EMBEDDING="cohere:embed-english-v3.0"

Fireworks

FIREWORKS_API_KEY=[Your key]
base_url="https://api.fireworks.ai/inference/v1/completions"
FAST_LLM="fireworks:accounts/fireworks/models/mixtral-8x7b-instruct"
SMART_LLM="fireworks:accounts/fireworks/models/mixtral-8x7b-instruct"
STRATEGIC_LLM="fireworks:accounts/fireworks/models/mixtral-8x7b-instruct"

EMBEDDING="fireworks:nomic-ai/nomic-embed-text-v1.5"

Bedrock

FAST_LLM="bedrock:anthropic.claude-3-sonnet-20240229-v1:0"
SMART_LLM="bedrock:anthropic.claude-3-sonnet-20240229-v1:0"
STRATEGIC_LLM="bedrock:anthropic.claude-3-sonnet-20240229-v1:0"

Other Embedding Models

Nomic

EMBEDDING="nomic:nomic-embed-text-v1.5"

VoyageAI

VOYAGE_API_KEY=[Your Key]
EMBEDDING="voyageai:voyage-law-2"