Skills

Managed Model Endpoints

Register a model service in the managed family, a local model server container the...

Install

npx skills add HughYau/AcademicForge --skill managed-model-endpoints -g -a claude-code -y
Repository
HughYau/AcademicForge2.6k
Category
Engineering
Updated
2mo ago
License
Apache-2.0
Context cost
Mediumabout 2,546 tokens
Runs scripts
No, instructions only

What it does

Register a model service in the managed family, a local model server container the daemon starts/stops on demand, or a remote upstream model API (https). Read the runbook, allocate a port (local only), compose idempotent start/stop scripts (local only), register once. Load when the user wants a model service available for inference, or when list_compute shows managed endpoints.

Managed model endpoints

A managed model endpoint is a model service the daemon owns: you register it once, then every compute_provider cell against it just works, the daemon swaps the resident model off the device (one model at a time, via the resident's own approved stop), runs your approved start script, waits for the readiness route, then runs your cell, streaming its lifecycle progress into the cell as it goes. You never run the container runtime yourself, never poll readiness in cells, and never see the credential value. Two verbs: register() (asks the user once) and ordinary inference cells.

Container specifics, image, registry login, internal port, cache mount target, readiness route, come from the model's own runbook skill; this skill is the translation contract.

Calling a registered endpoint, inference cells

Calling a registered endpoint, use the using-model-endpoint skill (this skill is the REGISTRATION contract; that one documents the call side in full).

The ONLY dispatch form is the compute_provider tool with the endpoint's registered name (list_compute shows them):

compute_provider(provider="boltz2-service", code="""
import requests
r = requests.post(BASE_URL + "/v1/infer", json=payload)
""")

The daemon brings the model up on demand (a first cold start downloads image + weights, minutes; let it run) and preloads BASE_URL into the cell, both as a Python variable (use it directly, as above) and as os.environ["BASE_URL"] (plus INFER_API_KEY for remote endpoints). Endpoints are not kernel environments: environment="boltz2-service" on a plain python cell fails, plain cells get no BASE_URL.

Enablement, once per machine

Read the whole SKILL.md on GitHub

More in Engineering