InferCrate

Your models, on your infrastructure

InferCrate

Open-source, self-hosted local LLM server.

Run local language models behind API-key authentication. Powered by Ollama and a Caddy authentication proxy, with persistent model storage, management helpers, and native Ollama and OpenAI-compatible APIs.

View on GitHub

Open source · MIT licensed

Get the book on Amazon

The Self-Hosted AI Builder's Guide Practical guidance for deployment, security, backups, and everyday operation.

Your application → your model

Choose a model.
Keep it close.

MODEL

llama3.2:3b

REQUEST

Summarize these notes in three bullet points.

  • Ollama API
  • OpenAI-compatible
  • Bearer auth
Illustrative local-model request. You select and download the model; quality, memory use, and speed depend on that choice.

A focused service for your applications

A local model server you can operate.

Serve models to your own applications, scripts, and compatible chat clients. InferCrate packages Ollama with access control and management helpers; your client provides the interface.

Images are automatically built and published through public GitHub Actions workflows. Powered by Ollama, Caddy.

Start with one model

Run locally.
Ask your first question.

This CPU example exposes the API only on the Docker host. The sample model is a starting point, not a hardware or answer-quality guarantee.

  • Docker on a supported Linux host
  • Sufficient RAM and disk space for the selected model
  • Internet access to download the image and model

Full setup guide on GitHub

For access from other computers, follow the maintained HTTPS, authentication, and network-exposure instructions.

  1. Start InferCrate

    docker run \
  --name ollama \
  --restart=always \
  -v ollama-data:/var/lib/ollama \
  -p 127.0.0.1:11434:11434 \
  -d hwdsl2/ollama-server

    Use docker logs ollama to check initial startup before running the management commands.

  2. Get your key and pull a model

    infer_api_key="$(docker exec ollama ollama_manage --getkey)"
docker exec ollama ollama_manage --pull llama3.2:3b

    Wait for the model download to finish. The mounted data volume retains the generated key and downloaded model.

  3. Make a chat request

    curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Authorization: Bearer $infer_api_key" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.2:3b","messages":[{"role":"user","content":"Hello!"}]}'

    Run in the same shell as the key command. Model responses are generated locally when you select a local model.

Part of a larger toolkit

Build your Self-Hosted AI Stack

InferCrate is also available as part of Self-Hosted AI Stack. Deploy local models, chat, document processing, voice, and AI tools together, or choose a smaller configuration for your needs.

Explore Self-Hosted AI Stack

Connect the services you need

Route local and cloud model APIs

GatewayCrate adds a common model API and routing controls for configured providers. Use a local InferCrate endpoint, external providers, or an explicitly configured combination.

Explore GatewayCrate

Before you deploy

Is InferCrate free to use?

The InferCrate project is open source under the MIT License. You provide hardware or cloud hosting and pay associated costs. Models, dependencies, and external providers retain their own licenses and terms.

Does InferCrate include a chat website?

It is a model server with APIs. Use a compatible client or add a browser interface such as AnythingLLM through Self-Hosted AI Stack. This website does not offer hosted chat.

What is the relationship to Ollama?

InferCrate packages Ollama with a Caddy authentication proxy, persistent storage, and management helpers. Native model identifiers, the Ollama API, and technical container names remain unchanged.

Do I need an NVIDIA GPU?

No. You can run on CPU. Model size, context length, simultaneous requests, memory, and hardware determine performance. NVIDIA acceleration is an optional deployment path.

Will prompts stay on my server?

Requests to a local model are processed by your deployment. Check the model and client configuration before assuming every route is local; external models, tools, and services can send data elsewhere.

Can I use a downloaded model without internet?

Local inference can continue with the required model files available. Downloads, updates, external integrations, and optional usage counts can still require network access. Review your complete configuration for disconnected operation; opt out of counts with OLLAMA_DISABLE_USAGE_COUNTS=1.

The image includes optional aggregate usage counts. See the usage-count documentation; set OLLAMA_DISABLE_USAGE_COUNTS=1 to disable them.

Choose your model. Decide where its requests run.

View on GitHub