langchain.text_splitter.Tokenizer¶
class langchain.text_splitter.Tokenizer(chunk_overlap: 'int', tokens_per_chunk: 'int', decode: 'Callable[[list[int]], str]', encode: 'Callable[[str], List[int]]')[source]¶
Attributes
chunk_overlap
tokens_per_chunk
decode
encode
Methods
__init__(chunk_overlap, tokens_per_chunk, ...)
__init__(chunk_overlap: int, tokens_per_chunk: int, decode: Callable[[list[int]], str], encode: Callable[[str], List[int]]) → None¶