Search Engines

Search Engines are used to find the most relevant web sources and content for a given research task. You can specify your preferred web search or use any custom retriever of your choice.

Web Search Engines

GPT Researcher defaults to using the Tavily search engine for retrieving search results. But you can also use other search engines by specifying the RETRIEVER env var. Please note that each search engine has its own API Key requirements and usage limits.

For example:

RETRIEVER=bing

You can also specify multiple retrievers by separating them with commas. The system will use each specified retriever in sequence. For example:

RETRIEVER=tavily,arxiv

For academic literature reviews, combine a broad web retriever with scholarly retrievers:

RETRIEVER=tavily,openalex,semantic_scholar

Thanks to our community, we have integrated the following web search engines and research retrievers:

• Tavily - Default
• Bing - Env: RETRIEVER=bing
• Brave Search - Env: RETRIEVER=brave and BRAVE_API_KEY
• GroundRoute - Env: RETRIEVER=groundroute and GROUNDROUTE_API_KEY
• BoCha - Env: RETRIEVER=bocha and BOCHA_API_KEY
• Google - Env: RETRIEVER=google
• SearchApi - Env: RETRIEVER=searchapi
• Serp API - Env: RETRIEVER=serpapi
• Serper - Env: RETRIEVER=serper - Setup Guide
• Searx - Env: RETRIEVER=searx
• Duckduckgo - Env: RETRIEVER=duckduckgo
• Arxiv - Env: RETRIEVER=arxiv
• OpenAlex - Env: RETRIEVER=openalex; optional OPENALEX_EMAIL and OPENALEX_API_KEY
• Semantic Scholar - Env: RETRIEVER=semantic_scholar
• Exa - Env: RETRIEVER=exa
• fastCRW - Env: RETRIEVER=crw
• PubMedCentral - Env: RETRIEVER=pubmed_central
• Xquik - Env: RETRIEVER=xquik and XQUIK_API_KEY
• MCP - Env: RETRIEVER=mcp

Other search providers can be installed as separate packages. See Retriever Plugins.

Custom Retrievers

You can also use any custom retriever of your choice by specifying the RETRIEVER=custom env var. Custom retrievers allow you to use any search engine that provides an API to retrieve documents and is widely used for enterprise research tasks.

In addition to setting the RETRIEVER env, you also need to set the following env vars:

• RETRIEVER_ENDPOINT: The endpoint URL of the custom retriever.
• Additional arguments required by the retriever should be prefixed with RETRIEVER_ARG_ (e.g., RETRIEVER_ARG_API_KEY).

Example

RETRIEVER=custom
RETRIEVER_ENDPOINT=https://api.myretriever.com
RETRIEVER_ARG_API_KEY=YOUR_API_KEY

Response Format

For the custom retriever to work correctly, the response from the endpoint should be in the following format:

[
  {
    "url": "http://example.com/page1",
    "raw_content": "Content of page 1"
  },
  {
    "url": "http://example.com/page2",
    "raw_content": "Content of page 2"
  }
]

The system assumes this response format and processes the list of sources accordingly.

Search Engine Configuration

Brave Search

To use Brave Search as your search engine:

1. Get your API key from Brave Search API
2. Set the required environment variables:

RETRIEVER=brave
BRAVE_API_KEY=your_api_key_here

Serper

To use Serper as your search engine:

1. Get your API key from serper.dev
2. Set the required environment variables:

RETRIEVER=serper
SERPER_API_KEY=your_api_key_here

Optional Configuration:

SERPER_REGION=us                    # Country code (us, kr, jp, etc.)
SERPER_LANGUAGE=en                  # Language code (en, ko, ja, etc.)
SERPER_TIME_RANGE=qdr:w            # Time filter (qdr:h, qdr:d, qdr:w, qdr:m, qdr:y)
SERPER_EXCLUDE_SITES=youtube.com   # Exclude sites (comma-separated)

OpenAlex

To use OpenAlex for scholarly works:

RETRIEVER=openalex
OPENALEX_EMAIL=you@example.com      # Optional but recommended for polite API usage
OPENALEX_API_KEY=your_api_key_here  # Optional; improves rate limits

OpenAlex works without an API key, but setting an email or free API key can make larger research runs more reliable.

PubMed Central

To use PubMed Central full-text retrieval:

RETRIEVER=pubmed_central
NCBI_API_KEY=your_api_key_here      # Optional; improves NCBI rate limits
PUBMED_DB=pmc                       # Optional; defaults to pmc

Missing a retriever? Feel free to contribute to this project by submitting issues or pull requests on our GitHub page.