Context Length
#4
by
Sao10K
- opened
Hmm, is the decreased context length (8k tokens from model config) a side effect of the training done after pruning? This would have been pretty nice if it retained its long context abilities.
This is our plan, to work on longer context. We will update as soon as get results. Hopefully will be able to release the model with 128k context.
pmolchanov
changed discussion status to
closed