Models and inference

How an outpost gets models

Two sources: a hosted provider reached over HTTPS with an API key, and models the outpost serves itself on its CPU, its GPU or a rented GPU pool. Both appear in one catalog of model names that chat, the agent and jobs share. Auto, the default, routes each call to a model here or on a verified peer in the mesh: responsive hosted routes first, then each outpost’s preferred model, the fastest to accept first within each tier, with a route that failed recently tried last. A call fails over when its route breaks. An Auto turn with no reachable model waits for one, for up to an hour, rather than failing; a turn that names a model nothing holds is refused.

Connect a provider

The first drkm start, or drkm models setup at any time, offers three choices: connect an inference provider with an API key, run a local model sized for this device’s GPU, or set up later. The provider path takes an HTTPS base URL of any OpenAI-compatible endpoint, the key, typed hidden, and a model ID, then sends a short test completion before saving the key encrypted. One hosted provider per outpost.

Administrators can also do this in the console under Settings → Chat: Default model picks the model for new chats, Context windows shows each model’s window and where the figure came from, and Your own API key saves the provider connection with an optional context window and private CA.

drkm start
drkm models setup

drkm console models        # the chat catalog: models, served, holders, default

Run a model on your hardware

The wizard’s local choice supports Linux NVIDIA GPUs: it sizes a pinned open model to the GPU it finds and launches the packaged serving runtime, which needs NVIDIA drivers, Docker and the NVIDIA Container Toolkit, 80% of one GPU free and tens of gigabytes of disk for the first build. Outside the wizard, a Mac serves a language model through its native engine with drkm engine install, and a CPU-only Linux host serves a quantized GGUF model with drkm serve <name> --engine llama-cpp and an explicit engine profile.

The host commands stage a model into the outpost’s registry and serve it: drkm pull imports a snapshot by name or from a hub, drkm serve starts an engine on it, and drkm models list shows what the registry holds.

drkm models list
drkm pull hf://<org>/<repo> --name my-model --revision <commit>
drkm serve my-model --engine vllm      # or llama-cpp, mlx, detector

# Mac: serve a language model through the native engine, persistently
drkm engine install my-model
drkm engine remove

The Darkmatter catalog

Settings → Models lists the models Darkmatter publishes for outposts: the vision detectors, Overwatch (planes, ships, vehicles and infrastructure in satellite imagery) and Silhouette (people in drone footage), and a short pinned list of open language models that fit this outpost’s GPU. Each row shows where it runs and its state, and offers Deploy or Download and load, Stop and Remove; a model outside the list reaches an outpost only through the host commands above. Everyone can read the page; administrators act.

Deploy offers This node · GPU, This node · CPU or, under Advanced, a Modal pool of rented GPUs with a type, a count and how many to keep warm; the first pooled model adopts the pool and removing the last releases it. An installed outpost downloads catalog models with its update grant; an outpost that is not signed in shows a lock on the rows it lacks.

drkm start --detector            # serve the Overwatch detector here
drkm start --no-detector         # stop serving it

# Rent a GPU pool for the detectors from the host
drkm pool adopt modal --model overwatch-obb-r3-int8 --gpus 4 --warm 0
drkm pool release

Choose a model in chat

The composer’s Model menu lists Auto first, then every catalog name; a model that peers also hold shows here +N or the holder’s callsign. Below it, Effort runs from faster to smarter in four stops, or Auto. A new chat starts on the outpost’s default; reopening a conversation keeps its model. Each reply says which model answered.

drkm console chat --model auto --effort medium -p "Summarize today's jobs"

Models for jobs

A worker declares the models it needs by alias in package() or .map_batches(): a catalog name pins that model, a requirement such as {"kind": "agent", "intelligence": "medium"} states a floor that placement resolves, and {"kind": "completion", "model": "vision"} selects a configured completion lane. Inside the attempt, context.models["alias"] is a drkm.Model with predict(), predict_batch() and analyze_image(). A predict job needs no worker at all: it runs a served detector over each record’s one image, and the platform keeps every GPU the outpost has busy, pooled ones included.

1
2
3
4
5
6
7
8
9
10import drkm

drkm.connect()
records = [{"id": "harbor-01", "files": ["tiles/harbor-01.png"]}]

job = (drkm.parallelize(records)
       .predict("overwatch-obb-r3-int8", conf=0.5)
       .submit(name="Vessels in harbor imagery"))
job.wait()
print(job.rows())

Serve across the mesh

An administrator can ask another outpost to serve a catalog model by name, one of the vision detectors or an entry of that member’s pinned language list, never an arbitrary download. The member’s own rules decide: a detector must fit its free GPU memory, a language model starts only on an idle engine, nothing replaces what already serves, and another outpost stops only what it started.

drkm console peer-models <callsign>
drkm console request-model <callsign> silhouette-a2-v1-fp16
drkm console stop-model <callsign> silhouette-a2-v1-fp16