OctoAI: Create self-generating AI with micro models

OctoAI (formerly known as OctoML) is an AI infrastructure platform designed to help developers and enterprises run, tune, and scale generative AI models efficiently. Founded in 2019 by the creators of the Apache TVM open-source compiler, the company focuses on optimizing machine learning models to reduce inference costs and latency.

When OctoML launched in 2019, its primary focus was on optimizing machine learning (ML) models. The company gained recognition for its ability to fine-tune models and package them into deployable containers for different hardware setups.

This approach was well-received by ML engineers, but OctoML realized that there was a need for a more comprehensive solution to address the challenges faced by businesses when deploying ML-based applications.

OctoAI micro models

OctoML announced the launch of OctoAI, its latest offering that takes the company's services to the next level. While not a complete pivot, OctoAI represents a shift in emphasis from optimizing models to helping businesses leverage existing open-source models and customize them according to their specific needs.

The platform aims to simplify the process of building and deploying ML-based applications without the burden of managing the underlying infrastructure.

OctoAI micro models (now part of NVIDIA's portfolio) are specialized, small-footprint AI models and containers designed for efficient, fast inference on specific tasks. They are designed to run on edge devices and GPU-accelerated infrastructure, offering high performance for focused applications rather than general-purpose tasks.

Simplifying AI compute with OctoAI

The core idea behind OctoAI is to provide a self-optimizing compute service for AI. By leveraging OctoAI, businesses can build ML-based applications and put them into production without worrying about the complexities of ML infrastructure.

The platform offers a managed compute service that automates the selection of hardware based on user priorities, whether it's prioritizing low latency or cost efficiency. OctoAI also automatically optimizes models, resulting in cost savings and performance gains.

Multiple models, singular goal

While OctoAI provides automated optimizations, it also gives users the flexibility to set their own parameters and choose the hardware that best suits their requirements. However, OctoML expects that most users will find value in allowing OctoAI to manage these tasks, enabling them to focus on their core ML applications.

In addition, OctoML offers accelerated versions of popular foundation models such as Dolly 2, Whisper, FILM, FLAN-UL2, and Stable Diffusion. These pre-accelerated models are ready to use out of the box, saving businesses time and effort in implementing them.

The program is in early access now, but you may sign-up for it via the link here.

Notably, OctoML achieved impressive improvements with the Stable Diffusion model, making it three times faster and reducing costs by five times compared to the original model.