LongNet

ThoughtStorms Wiki

A LanguageModel with up to a billion tokens in its context window!!!!

https://arxiv.org/abs/2307.02486

https://github.com/microsoft/torchscale

Interesting overview. Points out it's 'Sparse Attention'

Backlinks (1 items)