Skip to main content

Configure LLM

As described in the introduction, the default LLM is OpenAI due to its superior performance and speed. With that said, GPT Researcher supports various open/closed source LLMs, and you can easily switch between them by adding the LLM_PROVIDER env variable and corresponding configuration params. Current supported LLMs are openai, google (gemini), azureopenai, ollama, anthropic, mistral, huggingface and groq.

Using any model will require at least updating the LLM_PROVIDER param and passing the LLM provider API Key. You might also need to update the SMART_LLM_MODEL and FAST_LLM_MODEL env vars. To learn more about support customization options see here.

Please note: GPT Researcher is optimized and heavily tested on GPT models. Some other models might run intro context limit errors, and unexpected responses. Please provide any feedback in our Discord community channel, so we can better improve the experience and performance.

Below you can find examples for how to configure the various supported LLMs.

Custom OpenAI

Create a local OpenAI API using llama.cpp Server.

Custom OpenAI API LLM

# use a custom OpenAI API LLM provider
LLM_PROVIDER="openai"

# set the custom OpenAI API url
OPENAI_BASE_URL="http://localhost:1234/v1"
# set the custom OpenAI API key
OPENAI_API_KEY="Your Key"

# specify the custom OpenAI API llm model
FAST_LLM_MODEL="gpt-3.5-turbo-16k"
# specify the custom OpenAI API llm model
SMART_LLM_MODEL="gpt-4o"

Custom OpenAI API Embedding

# use a custom OpenAI API EMBEDDING provider
EMBEDDING_PROVIDER="custom"

# set the custom OpenAI API url
OPENAI_BASE_URL="http://localhost:1234/v1"
# set the custom OpenAI API key
OPENAI_API_KEY="Your Key"

# specify the custom OpenAI API embedding model
OPENAI_EMBEDDING_MODEL="custom_model"

Azure OpenAI

EMBEDDING_PROVIDER="azureopenai"
AZURE_OPENAI_API_KEY="Your key"

Ollama

GPT Researcher supports both Ollama LLMs and embeddings. You can choose each or both. To use Ollama you can set the following environment variables

# Use ollama for both, LLM and EMBEDDING provider
LLM_PROVIDER=ollama

# Ollama endpoint to use
OLLAMA_BASE_URL=http://localhost:11434

# Specify one of the LLM models supported by Ollama
FAST_LLM_MODEL=llama3
# Specify one of the LLM models supported by Ollama
SMART_LLM_MODEL=llama3
# The temperature to use, defaults to 0.55
TEMPERATURE=0.55

Optional - You can also use ollama for embeddings

EMBEDDING_PROVIDER=ollama
# Specify one of the embedding models supported by Ollama
OLLAMA_EMBEDDING_MODEL=nomic-embed-text

Groq

GroqCloud provides advanced AI hardware and software solutions designed to deliver amazingly fast AI inference performance. To leverage Groq in GPT-Researcher, you will need a GroqCloud account and an API Key. (NOTE: Groq has a very generous free tier.)

Sign up

Update env vars

And finally, you will need to configure the GPT-Researcher Provider and Model variables:

# To use Groq set the llm provider to groq
LLM_PROVIDER=groq
GROQ_API_KEY=[Your Key]

# Set one of the LLM models supported by Groq
FAST_LLM_MODEL=Mixtral-8x7b-32768

# Set one of the LLM models supported by Groq
SMART_LLM_MODEL=Mixtral-8x7b-32768

# The temperature to use defaults to 0.55
TEMPERATURE=0.55

NOTE: As of the writing of this Doc (May 2024), the available Language Models from Groq are:

  • Llama3-70b-8192
  • Llama3-8b-8192
  • Mixtral-8x7b-32768
  • Gemma-7b-it

Anthropic

Anthropic is an AI safety and research company, and is the creator of Claude. This page covers all integrations between Anthropic models and LangChain.

LLM_PROVIDER=anthropic
ANTHROPIC_API_KEY=[Your key]

You can then define the fast and smart LLM models for example:

FAST_LLM_MODEL=claude-2.1
SMART_LLM_MODEL=claude-3-opus-20240229

You can then define the fast and smart LLM models for example:

FAST_LLM_MODEL=claude-2.1
SMART_LLM_MODEL=claude-3-opus-20240229

Mistral

Sign up for a Mistral API key. Then update the corresponding env vars, for example:

LLM_PROVIDER=mistral
ANTHROPIC_API_KEY=[Your key]
FAST_LLM_MODEL=open-mistral-7b
SMART_LLM_MODEL=mistral-large-latest

Together AI

Together AI offers an API to query 50+ leading open-source models in a couple lines of code. Then update corresponding env vars, for example:

LLM_PROVIDER=together
TOGETHER_API_KEY=[Your key]
FAST_LLM_MODEL=meta-llama/Llama-3-8b-chat-hf
SMART_LLM_MODEL=meta-llama/Llama-3-70b-chat-hf

HuggingFace

This integration requires a bit of extra work. Follow this guide to learn more. After you've followed the tutorial above, update the env vars:

LLM_PROVIDER=huggingface
HUGGINGFACE_API_KEY=[Your key]
FAST_LLM_MODEL=HuggingFaceH4/zephyr-7b-beta
SMART_LLM_MODEL=HuggingFaceH4/zephyr-7b-beta

Google Gemini

Sign up here for obtaining a Google Gemini API Key and update the following env vars:

Please make sure to update fast and smart models to corresponding valid Gemini models.

LLM_PROVIDER=google
GEMINI_API_KEY=[Your key]