oMLX vs. BaseRT

oMLX is a dedicated MLX server optimized for macOS, which significantly boosts the speed and efficiency of local AI tasks on Apple Silicon hardware. It specifically addresses the needs of coding agents by employing paged SSD KV caching, allowing cache blocks to be retained on disk; thus, previously accessed prefixes can be swiftly retrieved across various requests and even after server restarts, negating the need for recalculation from the ground up. Consequently, the duration required to produce the first token in extensive contexts can drop dramatically, from a span of 30 to 90 seconds down to under five seconds following the initial interaction. The server skillfully handles multiple requests simultaneously through a constant batching approach using mlx-lm’s BatchGenerator, which improves overall generation throughput by preventing requests from queuing behind a single task. oMLX can serve a diverse array of models concurrently, including LLMs, vision-language models, embedding models, and rerankers, while efficiently managing memory limitations through LRU eviction. Additionally, it supports any MLX-format model available from Hugging Face, including Qwen, LLaMA, Mistral, Gemma, DeepSeek, MiniMax, and GLM, and has the capability to work with models stored in the regular Hugging Face cache, directories linked to LM Studio, or any custom storage solutions, thus providing a seamless experience for users. This adaptability in model integration not only enhances the functionality of oMLX but also significantly benefits developers and researchers, making it a practical tool in various AI applications. Overall, oMLX stands out as a robust solution for maximizing the potential of AI on macOS systems.

BaseRT provides a powerful inference runtime for large language models, specifically tailored for Apple Silicon, enabling developers to effortlessly access models from Hugging Face, engage in local dialogue, or use an OpenAI-compatible API all through a single command-line interface. Boosted by expertly designed Metal kernels, BaseRT is engineered to excel in prefill and decoding efficiency on M-series Macs, with benchmark tests demonstrating performance that is up to 6.4 times faster in prefill tasks than llama.cpp, 3.9 times quicker than MLX, and achieving a decoding speed that surpasses competitors by 1.33 times. The BaseRT CLI is equipped to handle various tasks including model downloading, conversion, interactive chatting, serving functionalities, completion generation, benchmarking, inspection, and bundle signing. Its comprehensive server capabilities include chat interactions, text completions, embeddings, transcription services, tool calls, continuous batching, paged key-value caching, and prefix caching, while supporting models that process text, vision, and audio data. BaseRT utilizes a unique .base model format that features Q2–Q8 affine quantization, optional AWQ calibration, and signed bundles, and it can convert GGUF, Hugging Face, and MLX checkpoints seamlessly. In addition to these features, this groundbreaking runtime is specifically designed to harness the full potential of Apple Silicon, establishing itself as an indispensable resource for developers working in the AI domain. With its impressive efficiency and broad functionality, BaseRT stands out as a key innovation for the future of AI development on Apple platforms.