processing.text

Text processing functions

split_text

def split_text(text: str,
               max_length: int = 8192) -> Generator[str, None, None]

Split text into chunks of a maximum length

Arguments:

• text str - The text to split
• max_length int, optional - The maximum length of each chunk. Defaults to 8192.

Yields:

• str - The next chunk of text

Raises:

• ValueError - If the text is longer than the maximum length

summarize_text

def summarize_text(url: str,
                   text: str,
                   question: str,
                   driver: Optional[WebDriver] = None) -> str

Summarize text using the OpenAI API

Arguments:

• url str - The url of the text
• text str - The text to summarize
• question str - The question to ask the model
• driver WebDriver - The webdriver to use to scroll the page

Returns:

• str - The summary of the text

scroll_to_percentage

def scroll_to_percentage(driver: WebDriver, ratio: float) -> None

Scroll to a percentage of the page

Arguments:

• driver WebDriver - The webdriver to use
• ratio float - The percentage to scroll to

Raises:

• ValueError - If the ratio is not between 0 and 1

create_message

def create_message(chunk: str, question: str) -> Dict[str, str]

Create a message for the chat completion

Arguments:

• chunk str - The chunk of text to summarize
• question str - The question to answer

Returns:

Dict[str, str]: The message to send to the chat completion

write_to_file

def write_to_file(filename: str, text: str) -> None

Write text to a file

Arguments:

• text str - The text to write
• filename str - The filename to write to