LongLLaMA vs. GLM-5.3-Flash

This repository presents the research preview for LongLLaMA, an innovative large language model capable of handling extensive contexts, reaching up to 256,000 tokens or potentially even more. Built on the OpenLLaMA framework, LongLLaMA has been fine-tuned using the Focused Transformer (FoT) methodology. The foundational code for this model comes from Code Llama. We are excited to introduce a smaller 3B base version of the LongLLaMA model, which is not instruction-tuned, and it will be released under an open license (Apache 2.0). Accompanying this release is inference code that supports longer contexts, available on Hugging Face. The model's weights are designed to effortlessly integrate with existing systems tailored for shorter contexts, particularly those that accommodate up to 2048 tokens. In addition to these features, we provide evaluation results and comparisons to the original OpenLLaMA models, thus offering a thorough insight into LongLLaMA's effectiveness in managing long-context tasks. This advancement marks a significant step forward in the field of language models, enabling more sophisticated applications and research opportunities.

GLM-5.3-Flash is an efficiency-focused multimodal AI model from Z.ai that combines advanced reasoning, coding, agentic execution, and visual intelligence. It is the first GLM-5-series model designed with native multimodal capabilities, allowing it to work directly with both textual and visual inputs. The architecture uses 320 billion total parameters while activating only 18 billion at a time, significantly reducing the amount of computation required for inference. A hybrid attention design blends linear attention for local information with sparse attention for retrieving important context from much larger inputs. Z.ai also uses technologies such as IndexPool and Manifold-Constrained Hyper-Connections to improve memory efficiency, latency, and model scaling. The model can operate with context windows of up to one million tokens, making it suitable for large repositories, lengthy documents, extended agent sessions, and complex multimodal workflows. GLM-5.3-Flash was trained on a 30-trillion-token multimodal corpus intended to strengthen reasoning across code, images, interfaces, documents, spreadsheets, presentations, and other business artifacts. In software development scenarios, the model can visually inspect rendered applications, evaluate its own output, and iteratively correct layout, functionality, or interaction issues. Z.ai’s reported benchmark results show large improvements over GLM-5.2 in areas such as software engineering and automation, while placing GLM-5.3-Flash close to leading frontier systems on several coding and agentic evaluations. Before its formal release, the model was anonymously tested under the name ox-alpha on OpenCode and OpenRouter, where Z.ai says it became one of the most widely used models during its testing period. GLM-5.3-Flash is available through Z.ai’s API and coding products as well as through downloadable weights on Hugging Face, with deployment support for SGLang, vLLM, and TokenSpeed.