Local models

Local runtimes are providers like any other: list the ones the device has in providers and send the device's limits. The answer names the model as the runtime expects it, says what it needs, and is free to run.

Runtimes

GET /v1/vocabulary flags them local: true. The catalog offers a reviewed list of about twenty popular open models (Qwen, Llama, Gemma, Mistral, gpt-oss, DeepSeek distills, Phi), only those with a quality score, refreshed weekly. Ollama's tags and sizes are curated by hand, since Ollama has no listing API for its library.

Fitting the device

• max_memory_gb: the memory the model may use. Send what the device can spare, not its total: the OS and your app need some.
• max_vram_gb: GPU memory, for a full offload. On Apple silicon memory is shared, so memory and VRAM are the same figure.
• max_disk_gb: free disk for the download.

GB are 10⁹ bytes, so a "16 GB" machine (17.2 × 10⁹ bytes) has a little headroom. The limits never apply to cloud providers in the same request: they run elsewhere.

Send max_memory_gb or max_vram_gb whenever you list a local runtime. A request that lists local runtimes with cloud providers and neither limit is refused with 400 invalid_request on max_memory_gb: otherwise an app that copies every provider from /v1/vocabulary would get a free local model of any size. When only local runtimes are listed, the answer comes with a no_device_limits warning instead, as the model may not fit.

How memory is estimated

Memory is the quantized weights (the file's size), plus the KV cache (context × 2 × layers × kv_heads × head_dim × 2 bytes, with grouped-query attention), plus about 10% runtime overhead. It is stated at 8,192 tokens of context. A larger min_context adds the KV cache for each further token, so a long context can push a model over your limit. WebLLM states its own VRAM figure at its fixed context, which is used as it is.

Each local answer carries requirements: memory_gb, vram_gb and disk_gb at context_for_memory (your min_context when it is larger), and kv_bytes_per_token; plus quantization, parameters_b and the weights' license.

Quantization and quality

Lighter quantizations lose a little quality, so Rhumb takes a penalty off the model's score (from llama.cpp's perplexity measurements on Llama 3 8B). A penalised score is marked estimated, and the answer's quality.quantization_penalty says by how much.

MLX's 4-bit and WebLLM's q4f16_1 take the 4-bit row; weights released in their own format (gpt-oss in MXFP4) and 16-bit weights lose nothing. Within one model the lighter quantization that fits wins, since every local offering costs the same: nothing.

Budget, origin and everything else

Local models count as the cheap band, and their typical call is 0. Every other rule applies as for the cloud: origin follows the model's maker, so a local Qwen is cn; needs, license and min_quality (after the penalty) work as usual. When nothing fits, no_candidate counts what max_memory_gb and the other limits removed.

// Resolve a local model that fits the device: the runtimes it has installed
// as providers, and its memory as max_memory_gb. Local offerings are free;
// the answer says what the model needs (memory at the context you asked for,
// GPU memory, disk) and its quantization. Cache it for 24 hours.
//
//   RHUMB_KEY=nnm_live_... npx tsx local.ts
//
// RHUMB_URL overrides the base URL (default https://rhumbdata.com).
// With no RHUMB_KEY, or on any error, it returns the fallback.

import { totalmem } from "node:os";
import { pathToFileURL } from "node:url";

export type LocalModel = {
  provider: string;
  provider_model_id: string;
  quantization?: string;
  requirements?: { memory_gb: number; disk_gb?: number };
};

/**
 * Memory the model may use, in GB (10^9 bytes): three quarters of the
 * machine's, leaving room for the OS and your app.
 */
export function deviceMemoryGb(share = 0.75): number {
  return Math.floor((totalmem() / 1e9) * share);
}

export async function resolveLocalModel(
  query: Record<string, string>,
  fallback: LocalModel,
): Promise<LocalModel> {
  const key = process.env.RHUMB_KEY;
  if (!key) return fallback;
  const base = process.env.RHUMB_URL ?? "https://rhumbdata.com";
  const qs = new URLSearchParams(query).toString();
  try {
    const res = await fetch(`${base}/v1/resolve?${qs}`, {
      headers: { Authorization: `Bearer ${key}` },
      signal: AbortSignal.timeout(2_000),
    });
    // 422 no_candidate: nothing on these runtimes fits the device.
    if (!res.ok) throw new Error(`resolver answered ${res.status}`);
    const { model } = (await res.json()) as { model: LocalModel };
    return model;
  } catch (error) {
    console.error("resolver unavailable, using the fallback:", error);
    return fallback;
  }
}

async function main() {
  // A 16 GB laptop with Ollama and LM Studio. On the device itself, send
  // String(deviceMemoryGb()) and the runtimes you find installed.
  const model = await resolveLocalModel(
    {
      task: "text.chat",
      providers: "ollama,lmstudio",
      optimize: "quality",
      max_memory_gb: "16",
      min_context: "8192",
    },
    { provider: "ollama", provider_model_id: "llama3.2:3b" },
  );
  console.log(
    JSON.stringify({
      provider: model.provider,
      provider_model_id: model.provider_model_id,
      quantization: model.quantization,
      memory_gb: model.requirements?.memory_gb,
    }),
  );
}

// Run directly (not imported): print the answer for an example device.
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href)
  await main();

POST /v1/plan takes the device too: "device": { "memory_gb": 16 } becomes max_memory_gb unless constraints set it.