oMLX vs. Locally AI

oMLX is a dedicated MLX server optimized for macOS, which significantly boosts the speed and efficiency of local AI tasks on Apple Silicon hardware. It specifically addresses the needs of coding agents by employing paged SSD KV caching, allowing cache blocks to be retained on disk; thus, previously accessed prefixes can be swiftly retrieved across various requests and even after server restarts, negating the need for recalculation from the ground up. Consequently, the duration required to produce the first token in extensive contexts can drop dramatically, from a span of 30 to 90 seconds down to under five seconds following the initial interaction. The server skillfully handles multiple requests simultaneously through a constant batching approach using mlx-lm’s BatchGenerator, which improves overall generation throughput by preventing requests from queuing behind a single task. oMLX can serve a diverse array of models concurrently, including LLMs, vision-language models, embedding models, and rerankers, while efficiently managing memory limitations through LRU eviction. Additionally, it supports any MLX-format model available from Hugging Face, including Qwen, LLaMA, Mistral, Gemma, DeepSeek, MiniMax, and GLM, and has the capability to work with models stored in the regular Hugging Face cache, directories linked to LM Studio, or any custom storage solutions, thus providing a seamless experience for users. This adaptability in model integration not only enhances the functionality of oMLX but also significantly benefits developers and researchers, making it a practical tool in various AI applications. Overall, oMLX stands out as a robust solution for maximizing the potential of AI on macOS systems.

Locally AI is a cutting-edge application that enables users to harness the power of advanced language models directly on their iPhones, iPads, or Macs without relying on cloud services or an internet connection. Utilizing Apple’s MLX framework, it offers rapid performance while maintaining low power consumption, which results in a seamless experience for chatting, creating, learning, and exploring AI functionalities across a variety of devices. The application accommodates a selection of open models, such as Llama, Gemma, Qwen, and DeepSeek, allowing users to effortlessly switch between them and tailor outputs for different tasks. Functioning entirely offline, it removes the necessity for logins and ensures that no data is collected or transmitted, thus providing complete privacy and control over personal information. Users can interact with AI through natural conversations, evaluate documents or images, and generate text through a user-friendly interface designed for simplicity and responsiveness. This thoughtful design not only fosters creativity and exploration but also significantly enriches the overall user experience, making it an invaluable tool for anyone looking to engage with AI. Ultimately, Locally AI empowers users to take full advantage of AI technology while prioritizing their privacy and ease of use.