LongNet

Cardigan Bay Wiki

A LanguageModel with up to a billion tokens in its context window!!!!

https://arxiv.org/abs/2307.02486

https://github.com/microsoft/torchscale

Interesting overview. Points out it's 'Sparse Attention'

Backlinks (1 items)

No Transclusions

Page Last Modified @ Mon Jul 10 19:08:43 BST 2023 Exported from CardiganBay @ 2026-01-03T20:20:41.750278374