Odd response dependent on file size
Ran into a bit of an odd limitation running this model locally. I'm passing it a file and asking it to perform different operations on it. It usually works until the file size reaches somewhere between 5500 - 7500 chars (haven't checked tokens). At that point, it practically ignores the question/task and just gives a generic/default answer breaking down the file into its components. Something similar to the following: This data represents .... Here's a breakdown of the information provided regardless of what question I posed in the prompt. Tried it with a few different files with the same results.
Any clues or ideas of what kinda limit am I hitting?
hi, what kinds of operations? completion or revision or some kind of understanding? what tools did you use to conduct the inference?
Hey! It doesn't matter what operation you ask it to perform once you hit that character limit cause it will respond the same way. You can ask it to redact personal information, ask it to summarize ... I'm on a Mac cli running Ollama locally along with this https://llm.datasette.io/en/stable/. Tried it with this model and the instruct one as well, with the same results.
I think I know what's going on. It's reaching the 2048 prompt limit and thus whatever instruction I'm giving it, it's not even seeing which appears as though it's ignoring it when in fact, it's not getting to it.
time=2024-11-14T13:15:54.403-06:00 level=WARN source=runner.go:126 msg="truncating input prompt" limit=2048 prompt=2390 numKeep=4
ollama by default uses a context length of 2048 tokens. you can change that by following https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-specify-the-context-window-size.