Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Strategies

• Comparing hosted APIs versus self-hosted solutions and their trade-offs.
• Patterns for on-premises, edge, and air-gapped implementations.
• Single model deployments versus managing a model portfolio.
• Evaluating cost efficiency per decision.

Hardware Sizing and Capacity Planning

• Calculating memory footprints based on parameter counts.
• Choosing between CPU-only execution and GPU acceleration.
• Planning for concurrency and throughput requirements.
• Allowing headroom for anticipated growth in task length.

Serving Infrastructure

• Evaluating vLLM and SGLang options.
• Executing models via llama.cpp and GGUF formats.
• Using Ollama for local development workflows.
• Selecting the optimal stack based on the specific shape of decision workloads.

Quantization While Preserving Calibration

• Reviewing precision formats and associated memory savings.
• Measuring impacts on model quality and calibration loss.
• Exploring per-tensor and importance-aware quantization variants.
• Conducting acceptance testing following the quantization process.

Latency Optimization

• Distinguishing between median and tail latency.
• Analyzing how latency scales with context length.
• Utilizing warm starts and model caching strategies.
• Batching multiple decision queries for efficiency.

Edge and Constrained Environments

• Running small decision models on CPU resources.
• Deploying on Apple Silicon and embedded systems.
• Managing offline operations and update strategies.
• Distributing model weights in air-gapped settings.

Containerization and Scaling

• Designing container images optimized for model serving.
• Managing volumes and cache effectively.
• Implementing horizontal scaling and routing logic.
• Setting resource limits and sharing GPUs efficiently.

Observability and Monitoring

• Tracking request volume, latency, and error rates.
• Monitoring confidence distributions.
• Logging decision outcomes and escalation events.
• Setting alerts for calibration drift.

Versioning and Rollout Management

• Managing versions of models and adapters.
• Implementing shadow and canary deployment strategies.
• Pinning model files for audit compliance.
• Establishing robust rollback procedures.

Security and Multi-Tenancy

• Implementing authentication and rate limiting.
• Ensuring tenant isolation for prompts and logs.
• Configuring network isolation and egress controls.
• Maintaining audit logs of all decisions.