Skip to main content

processing.text

Text processing functions

split_text​

def split_text(text: str,
               max_length: int = 8192) -> Generator[str, None, None]

Split text into chunks of a maximum length

Arguments:

  • text str - The text to split
  • max_length int, optional - The maximum length of each chunk. Defaults to 8192.

Yields:

  • str - The next chunk of text

Raises:

  • ValueError - If the text is longer than the maximum length

summarize_text​

def summarize_text(url: str,
                   text: str,
                   question: str,
                   driver: Optional[WebDriver] = None) -> str

Summarize text using the OpenAI API

Arguments:

  • url str - The url of the text
  • text str - The text to summarize
  • question str - The question to ask the model
  • driver WebDriver - The webdriver to use to scroll the page

Returns:

  • str - The summary of the text

scroll_to_percentage​

def scroll_to_percentage(driver: WebDriver, ratio: float) -> None

Scroll to a percentage of the page

Arguments:

  • driver WebDriver - The webdriver to use
  • ratio float - The percentage to scroll to

Raises:

  • ValueError - If the ratio is not between 0 and 1

create_message​

def create_message(chunk: str, question: str) -> Dict[str, str]

Create a message for the chat completion

Arguments:

  • chunk str - The chunk of text to summarize
  • question str - The question to answer

Returns:

Dict[str, str]: The message to send to the chat completion

write_to_file​

def write_to_file(filename: str, text: str) -> None

Write text to a file

Arguments:

  • text str - The text to write
  • filename str - The filename to write to