langchain.text_splitter.TextSplitter¶
class langchain.text_splitter.TextSplitter(chunk_size: int = 4000, chunk_overlap: int = 200, length_function: ~typing.Callable[[str], int] = <built-in function len>, keep_separator: bool = False, add_start_index: bool = False, strip_whitespace: bool = True)[source]¶
Interface for splitting text into chunks.
Create a new TextSplitter.
Parameters
chunk_size – Maximum size of chunks to return
chunk_overlap – Overlap in characters between chunks
length_function – Function that measures the length of given chunks
keep_separator – Whether to keep the separator in the chunks
add_start_index – If True, includes chunk’s start index in metadata
strip_whitespace – If True, strips whitespace from the start and end of
every document
Methods
__init__([chunk_size, chunk_overlap, ...])
Create a new TextSplitter.
atransform_documents(documents, **kwargs)
Asynchronously transform a sequence of documents by splitting them.
create_documents(texts[, metadatas])
Create documents from a list of texts.
from_huggingface_tokenizer(tokenizer, **kwargs)
Text splitter that uses HuggingFace tokenizer to count length.
from_tiktoken_encoder([encoding_name, ...])
Text splitter that uses tiktoken encoder to count length.
split_documents(documents)
Split documents.
split_text(text)
Split text into multiple components.
transform_documents(documents, **kwargs)
Transform sequence of documents by splitting them.