Local AI Proxy Setup Guide

Run any model — local or cloud — through one OpenAI-compatible URL that every tool in your stack can share. Cursor, your web app, your agents, your scripts: one endpoint, one key, all models.

What You're Building

Prerequisites

A Mac with Apple Silicon (M1/M2/M3/M4) or a Linux box with a GPU
Python 3.11+
Node.js 20+ (for Tailscale CLI and optional tooling)
A Tailscale account (free tier works)

Step 1: Install Ollama

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

Pull the models you want to serve locally:

ollama pull qwen3:8b            # good starter (8B, fast)
ollama pull bge-m3              # embeddings (1024-dim, multilingual)
ollama pull qwen3.6:27b-q8_0   # larger reasoning model (needs 32GB+ RAM)

Verify Ollama is running:

curl http://127.0.0.1:11434/api/tags | python3 -m json.tool

Step 2: Install LiteLLM

Create a dedicated directory and Python venv:

mkdir -p ~/ai-proxy && cd ~/ai-proxy
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip litellm

Step 3: Write Your Config

Create ~/ai-proxy/litellm.config.yaml:

# LiteLLM config — edit model_name entries to match what you want
# to call from Cursor / your apps.

model_list:
  # --- LOCAL MODELS (Ollama) ---

  # Your main local chat model
  - model_name: local-chat
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://127.0.0.1:11434
      keep_alive: \&#x22;24h\&#x22;
      max_tokens: 4096
      num_ctx: 8192

  # Embeddings
  - model_name: local-embed
    litellm_params:
      model: ollama/bge-m3:latest
      api_base: http://127.0.0.1:11434
      keep_alive: \&#x22;5m\&#x22;

  # --- CLOUD MODELS ---

  # DeepSeek V4 Pro (~$0.44/M input, $0.87/M output)
  - model_name: deepseek-v4-pro
    litellm_params:
      model: deepseek/deepseek-v4-pro
      api_key: os.environ/DEEPSEEK_API_KEY
      max_tokens: 16384

  # DeepSeek V4 Flash (~$0.04/M input, $0.09/M output)
  - model_name: deepseek-v4-flash
    litellm_params:
      model: deepseek/deepseek-v4-flash
      api_key: os.environ/DEEPSEEK_API_KEY
      max_tokens: 8192

  # MiMo V2.5 — Xiaomi 310B MoE, 1M context ($0.14/M in, $0.28/M out)
  - model_name: mimo-v2.5
    litellm_params:
      model: openai/mimo-v2.5
      api_base: https://api.xiaomimimo.com/v1
      api_key: os.environ/MIMO_API_KEY
      max_tokens: 16384

  # MiniMax M3 — 1M context ($0.14/M in, $0.28/M out)
  - model_name: minimax-m3
    litellm_params:
      model: openai/MiniMax-M3
      api_base: https://api.minimax.io/v1
      api_key: os.environ/MINIMAX_API_KEY
      max_tokens: 16384

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Step 4: Set Your Secrets

Create ~/.ai-proxy-keys (chmod 600):

Get your API keys from:

DeepSeek: https\://platform.deepseek.com/api\_keys
MiMo: https\://platform.xiaomimimo.com
MiniMax: https\://platform.minimaxi.com

The LITELLM_MASTER_KEY is any string you choose — it's the password your clients use to talk to the proxy. Make it strong; it gates access to all your models.

Step 5: Start the Proxy

cd ~/ai-proxy
source .venv/bin/activate
source ~/.ai-proxy-keys

.venv/bin/litellm --config litellm.config.yaml --port 4000

Test it:

# List available models
curl -s http://127.0.0.1:4000/v1/models \
  -H \&#x22;Authorization: Bearer $LITELLM_MASTER_KEY\&#x22; | python3 -m json.tool

# Chat with your local model
curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H \&#x22;Authorization: Bearer $LITELLM_MASTER_KEY\&#x22; \
  -H \&#x22;Content-Type: application/json\&#x22; \
  -d

Step 6: Expose It With Tailscale Funnel

This gives you a stable HTTPS URL reachable from anywhere — your phone, Vercel, another machine.

# Install Tailscale (if not already)
# macOS: brew install tailscale  OR  download from tailscale.com
# Linux: curl -fsSL https://tailscale.com/install.sh | sh

# Log in
tailscale up

# Expose your proxy to the internet via Funnel
tailscale funnel 4000

Tailscale prints your public URL, something like:

https://your-machine.tail-network.ts.net

This persists across reboots. Now your proxy is reachable from anywhere as:

https://your-machine.tail-network.ts.net/v1

Test from another machine:

curl -s https://your-machine.tail-network.ts.net/v1/models \
  -H \&#x22;Authorization: Bearer sk-your-chosen-password-here\&#x22;

Step 7: Connect Cursor IDE

In Cursor:

Settings → Models → scroll to OpenAI API Key
Paste your LITELLM_MASTER_KEY value
Set Override OpenAI Base URL to:

   https://your-machine.tail-network.ts.net/v1

(or http://127.0.0.1:4000/v1 if Cursor runs on the same machine)

Under Model Names, add your custom models:

local-chat
deepseek-v4-pro
deepseek-v4-flash
mimo-v2.5
minimax-m3

Now when you select any of those models in Cursor's model picker, it routes through your proxy to the right backend.

Step 8: Connect Your Web App (Vercel AI SDK)

import { createOpenAI } from

On Vercel, set these environment variables:

LOCAL_LLM_BASE_URL = https://your-machine.tail-network.ts.net/v1
LOCAL_LLM_API_KEY = your LITELLM_MASTER_KEY

Step 9: Make It Survive Reboots (macOS)

Create ~/Library/LaunchAgents/com.local.litellm.plist:

Load it:

# Replace YOU with your username in the plist first, then:
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.local.litellm.plist

For Linux, use a systemd unit instead.

Quick Reference

\| What | URL | Auth | |------|-----|------| | Local (same machine) | http://127.0.0.1:4000/v1 | Bearer <LITELLM_MASTER_KEY> | | Remote (Tailscale Funnel) | https://your-machine.tail-network.ts.net/v1 | Same | | List models | GET /v1/models | Same | | Chat | POST /v1/chat/completions | Same | | Embeddings | POST /v1/embeddings | Same | | Health check | GET /health/liveness | None |

Adding More Models Later

Just add a new entry to litellm.config.yaml and restart:

  - model_name: my-new-model
    litellm_params:
      model: openai/some-model-name     # \&#x22;openai/\&#x22; prefix for any OpenAI-compatible API
      api_base: https://api.provider.com/v1
      api_key: os.environ/PROVIDER_API_KEY
      max_tokens: 16384

For local Ollama models, use ollama_chat/ (chat) or ollama/ (embeddings) prefix:

  - model_name: my-local-llama
    litellm_params:
      model: ollama_chat/llama3.3:70b
      api_base: http://127.0.0.1:11434
      keep_alive: \&#x22;1h\&#x22;

No client code changes needed — just use the new model_name in your requests.

Troubleshooting

\| Symptom | Cause | Fix | |---------|-------|-----| | 401 Unauthorized | Wrong key in client | Use LITELLM_MASTER_KEY, not the provider key | | Model not found | Typo in model name | Must match model_name in YAML exactly | | Ollama timeout | Model not pulled | Run ollama pull <model> first | | Cloud 401/502 | Provider key missing | Check ~/.ai-proxy-keys has the right export | | Funnel unreachable | Tailscale not running | tailscale up then tailscale funnel 4000 | | Port already in use | Stale process | lsof -ti :4000 | xargs kill -9 then restart |

Why This Architecture

One endpoint for everything. Your app code doesn't know or care whether it's hitting a 7B model on your GPU or DeepSeek's cloud. Switch models by changing a string.
Zero vendor lock-in. Add a new provider by adding 5 lines of YAML. Drop one by removing them.
Cost control. Route cheap tasks to local models ($0), expensive reasoning to cloud ($0.001/turn). Your proxy, your rules.
Privacy. Local models never leave your machine. Cloud calls are opt-in per model name.
Works with everything. Any tool that speaks the OpenAI API spec (Cursor, Vercel AI SDK, LangChain, OpenAI Python client, curl) works unchanged.