dstack Prototyping

Use /dstack for CLI commands, YAML fields, apply/attach behavior, service URLs, and other dstack syntax. This skill explains how to use dstack runs while the model-serving configuration is still unknown.

Goal

Find a working dstack service configuration for the requested model.

Before submitting a service, use a task on real hardware to test the serving image, install/runtime assumptions, model download, cache path, command, port, launch flags, resources, env vars, backend/fleet choice, and local model request. Then submit the same configuration as a service and verify the model through the dstack service URL.

Choose Where To Run

Pick the offer whose hardware best fits the goal at hand. Only when several offers fit comparably, choose a VM-based backend, an SSH fleet, or a Kubernetes fleet: they support idle instances and/or instance volumes, so later runs reuse the provisioned/idle instance or instance volumes for caching model weights (and possibly other writes), while container-based backends start clean on every run.

Fetch https://dstack.ai/docs/concepts/backends.md and classify backends from the fetched document, not from memory.

If the intention is to use PD disaggregation, the fleet must use placement: cluster. Since PD disaggregation implies running a router, unlike workers that must run on GPUs, the router normally should run on a CPU instance. Use dstack fleet to see existing fleets and dstack fleet get <fleet name> --json to inspect a specific fleet.

Check Serving Sources

Check serving-framework sources early enough to choose the image, command, launch flags, resources, cache paths, request format, and expected model behavior.

For vLLM and SGLang, use these as credible sources: