Anthropic, the OpenAI competitor just announced a new version of their LLM Claude. This new Version has a context length of 100K tokens, which corresponds to around 75K words. It is not clear from the announcement how they implemented that and how the full context gets fed into the attention layers.

OpenAI is planning to release a 32K context version of GPT-4 soon.

Longer context means you can feed long-form content like books, reports, or entire code bases into the model and work with the entirety of the data.