SGLang

SGLang is a high-performance serving framework designed for low-latency and high-throughput inference of large language and multimodal models across various hardware setups, from single GPUs to large distributed clusters.

Quick Start

# Launch server (in terminal)
# python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000

# Use with OpenAI client
import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    temperature=0,
    max_tokens=64,
)
print(response.choices[0].message.content)

# Offline batch inference (no HTTP server)
import sglang as sgl

llm = sgl.Engine(model_path="meta-llama/Llama-3.1-8B-Instruct")

prompts = ["Hello, my name is", "The capital of France is"]
outputs = llm.generate(prompts, {"temperature": 0.8, "top_p": 0.95})

for prompt, output in zip(prompts, outputs):
    print(f"Prompt: {prompt}\nGenerated: {output['text']}\n")

Installation

# Method 1: pip/uv (recommended)
pip install --upgrade pip
pip install uv
uv pip install "sglang"

# Method 2: Docker
docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-token>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000

Documentation

Full documentation in docs/. See docs/000-index.md for detailed navigation.

By Topic

By Keyword

Learning Path

1. Foundation (001-003): Overview, installation
2. Basic Usage (004-037): API fundamentals, model-specific guides
3. Supported Models (038-059): Model architectures, extending support
4. Advanced Features (060-105): Structured outputs, optimization, scaling
5. Platform Guides (106-128): Hardware-specific deployment
6. Distributed Deployment (129-137): Multi-node, Kubernetes
7. Production (138-151): Monitoring, troubleshooting, frontend language
8. Development (152-169): Contributing, benchmarking

Common Tasks

Launch a Server

-> docs/004-basic-usage-send-request.md (Quick start guide for server + requests)

Use OpenAI-Compatible API

-> docs/005-basic-usage-openai-api-completions.md (Chat/text completions with OpenAI client)

Batch Inference Without Server

-> docs/020-basic-usage-offline-engine-api.md (Offline engine for batch processing)

Constrain Output with JSON Schema

-> docs/061-advanced-features-structured-outputs.md (JSON, regex, EBNF constraints)

Deploy DeepSeek Models

-> docs/012-basic-usage-deepseek-v3.md (DeepSeek V3/V3.1/R1 configuration)

Enable Quantization

-> docs/064-advanced-features-quantization.md (FP8/INT4/AWQ/GPTQ quantization)

Serve Multiple LoRA Adapters

-> docs/066-advanced-features-lora.md (Multi-LoRA batching)

Use Speculative Decoding

-> docs/067-advanced-features-speculative-decoding.md (EAGLE-based acceleration)

Configure HiCache (KV Caching)

-> docs/070-advanced-features-hicache-best-practices.md (Hierarchical caching setup)

Multi-Node Deployment

-> docs/132-references-multi-node-deployment-multi-node.md (Distributed inference)

Deploy on Kubernetes

-> docs/130-references-multi-node-deployment-deploy-on-k8s.md (K8s deployment guide)

Monitor with Prometheus

-> docs/141-references-production-metrics.md (Production metrics setup)

Troubleshoot Common Issues

-> docs/140-references-faq.md (CUDA OOM, common errors)

Tune Server Performance

-> docs/088-advanced-features-hyperparameter-tuning.md (Throughput optimization)

Deploy on AMD GPUs

-> docs/106-platforms-amd-gpu.md (ROCm/MI300 setup)