Create documents from a list of texts.
classmethod from_huggingface_tokenizer(tokenizer: Any, **kwargs: Any) → TextSplitter¶
Text splitter that uses HuggingFace tokenizer to count length.
classmethod from_language(language: Language, **kwargs: Any) → RecursiveCharacterTextSplitter[source]¶
classmethod from_tiktoken_encoder(encoding_name: str = 'gpt2', model_name: Optional[str] = None, allowed_special: Union[Literal['all'], AbstractSet[str]] = {}, disallowed_special: Union[Literal['all'], Collection[str]] = 'all', **kwargs: Any) → TS¶
Text splitter that uses tiktoken encoder to count length.
static get_separators_for_language(language: Language) → List[str][source]¶
split_documents(documents: Iterable[Document]) → List[Document]¶
Split documents.
split_text(text: str) → List[str][source]¶
Split text into multiple components.
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]¶
Transform sequence of documents by splitting them.
Examples using RecursiveCharacterTextSplitter¶
RePhraseQueryRetriever
Cohere Reranker
Ollama
Zep
your local model path
Loading documents from a YouTube url
Source Code
Set env var OPENAI_API_KEY or load from a .env file:
Set env var OPENAI_API_KEY or load from a .env file
Question Answering
Perform context-aware text splitting
Use local LLMs
QA using Activeloop’s DeepLake
!pip install bs4
MultiVector Retriever
MultiQueryRetriever
Parent Document Retriever
MarkdownHeaderTextSplitter