LongNet

ThoughtStorms Wiki

A LanguageModel with up to a billion tokens in its context window!!!!

https://arxiv.org/abs/2307.02486

https://github.com/microsoft/torchscale

Interesting overview. Points out it's 'Sparse Attention'

Backlinks (1 items)

No Transclusions

Page Last Modified @ Mon Jul 10 18:08:43 UTC 2023 Exported from CardiganBay @ 2026-06-24T13:30:18.234765741