How to Use LocalAI: Run ChatGPT-Style AI on Your Own Server

What It Is

LocalAI is an open-source AI engine that runs models on your own hardware: large language models, vision, voice, image, and video workloads, all behind one API. Its headline promise is that no GPU is required, so an ordinary computer or a modest server can serve AI features. The key design idea is a small core with backends pulled on demand. Each backend wraps a best-in-class engine, such as llama.cpp, vLLM, whisper.cpp, stable-diffusion, or MLX, in its own image, downloaded only when a model needs it. Perhaps most useful for a small business, LocalAI offers drop-in API compatibility with OpenAI, Anthropic, and ElevenLabs, so software that already talks to those APIs can often be pointed at LocalAI instead. On GitHub, the repository mudler/LocalAI has about 49,598 stars and is released under the MIT licence.

Who It’s For / SMB Use Case

LocalAI fits businesses that already use, or want to build, software powered by AI APIs and would rather run those calls in-house. A small agency with a quoting tool, a booking form, or an internal helper that calls the OpenAI API can move that traffic to a local server and cut per-token bills. A business handling sensitive client information can keep prompts and documents inside its own infrastructure. It also suits a technically curious owner who wants one server for several jobs: text generation, transcription, image generation, and voice features, instead of a separate subscription for each. You will need someone comfortable with Docker and the command line. If your needs are only occasional chat, a desktop app is simpler. LocalAI pays off when AI calls are frequent, repeated, or privacy-sensitive.

Installation

The README documents container installs for CPU and several GPU types, plus a macOS download. Docker or Podman is the main requirement. If you already ran LocalAI before, the README notes that docker start -i local-ai restarts an existing container.

CPU-only install (Docker)

docker run -ti --name local-ai -p 8080:8080 localai/localai:latest

GPU installs (Docker)

For an Nvidia GPU, the README lists CUDA 13 and CUDA 12 images, plus Jetson ARM64 builds:

# CUDA 13
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13

# CUDA 12
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12

For AMD, Intel, and Vulkan hardware, the README documents these commands:

# AMD GPU (ROCm)
docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-gpu-hipblas

# Intel GPU (oneAPI)
docker run -ti --name local-ai -p 8080:8080 --device=/dev/dri/card1 --device=/dev/dri/renderD128 localai/localai:latest-gpu-intel

# Vulkan GPU
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-vulkan

On macOS, the README links a DMG installer and warns that it is not signed by Apple. After installing, you may need to run:

sudo xattr -d com.apple.quarantine /Applications/LocalAI.app

Once the container is running, the API and web interface are available on port 8080. For full setup detail, see the Getting Started guide at localai.io.

How to Use

1. Load a model. The local-ai run command pulls a model from the gallery, Hugging Face, the Ollama registry, or an OCI registry. Start with a small instruct model, as shown in the README:

   local-ai run llama-3.2-1b-instruct:q4_k_m

   Other sources from the README include local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf and local-ai run ollama://gemma:2b. Browse available models with local-ai models list or at models.localai.io.

2. Chat from the terminal. With a server running in one terminal, open a second terminal and start the built-in chat:

   local-ai chat --model llama-3.2-1b-instruct:q4_k_m

   Inside a session, /models lists installed models and /model switches between them.

3. Call it from your app. Because LocalAI is OpenAI-compatible, point an existing OpenAI client at your server instead of the cloud API. The base URL is http://localhost:8080/v1. A minimal request looks like this:

   curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "llama-3.2-1b-instruct:q4_k_m", "messages": [{"role": "user", "content": "Draft a polite reply to a customer asking about delivery times."}]}'

   Check the official docs for the full endpoint list, since available endpoints vary by backend.

4. Try other modalities. The same server can handle transcription, image generation, text-to-speech, and more, depending on which backends and models you load. The README notes that LocalAI automatically detects your GPU capabilities and downloads the appropriate backend. Start with one modality, confirm it works, then add the next.

Pricing / Limitations

LocalAI is free, open-source software under the MIT licence, so there is no licence fee for the engine itself. Your real costs are the computer or server it runs on, electricity, and your time for setup and maintenance. Model downloads are generally free, but each model carries its own licence, so check the licence of any model you use commercially before putting it in front of customers. There is no per-token bill, which is the main financial advantage. The trade-offs are operational. You are responsible for updates, backups, access control, and security. Answer quality depends on the model you pick, and large models need plenty of RAM or VRAM. The README describes multi-user support with API key auth, user quotas, and role-based access, but configuring all of that takes more work than signing up for a hosted service. For production use, review the deployment docs rather than treating a quickstart container as finished.

Bottom Line for SMB

LocalAI is the best fit when your business already calls AI APIs and wants to bring those calls in-house. Start with the CPU Docker image and one small model, point a single tool at it, and compare the results and running costs with your current provider. If that test works, expand gradually to more models and workloads.

FAQ

Do I need a GPU to run LocalAI?

No. The README states that no GPU is required. CPU-only containers work, and GPU images for Nvidia, AMD, Intel, and Vulkan hardware are available if you want faster responses.

Can LocalAI replace the OpenAI API in my existing app?

In many cases, yes. LocalAI advertises drop-in API compatibility with OpenAI, Anthropic, and ElevenLabs. Point your client at your LocalAI server and test your existing calls, since exact endpoint coverage depends on the backend.

Where do I find models for LocalAI?

Browse the model gallery at models.localai.io, or run local-ai models list. Models can come from the gallery, Hugging Face, the Ollama registry, a YAML config, or a standard OCI registry.

Related reading: AI Agents Explained for Small Business and Per-Seat vs Usage-Based AI Pricing.