Together Dedicated Containers

Overview

Run custom Dockerized inference workloads on Together’s managed GPU infrastructure. You bring the container — Together handles compute, autoscaling, networking, and observability.

Components:

• Jig CLI: Build, push, and deploy containers
• Sprocket SDK: Python SDK for handling inference requests inside containers
• Container Registry: registry.together.xyz for storing images
• Queue API: Async job submission with priority and progress tracking

Workflow

1. Write inference code using Sprocket SDK (setup() + predict())
2. Build container with Jig CLI (jig build)
3. Push to registry (jig push)
4. Deploy (jig deploy)
5. Send requests to your deployment

Quick Start

1. Install Jig CLI

pip install jig-together
jig auth login  # Authenticate with Together AI

2. Create Inference Worker

# worker.py
from sprocket import SprocketWorker

class MyWorker(SprocketWorker):
    def setup(self):
        """Load model and resources (runs once at startup)."""
        import torch
        self.model = torch.load("model.pt")

    def predict(self, request):
        """Handle a single inference request."""
        input_data = request.json()
        result = self.model(input_data["prompt"])
        return {"output": result}

3. Configure Project

# pyproject.toml
[tool.jig]
name = "my-inference-service"
gpu = "A100"
gpu_count = 1
worker = "worker:MyWorker"

4. Build, Push, Deploy

jig build                    # Build Docker image
jig push                     # Push to registry.together.xyz
jig deploy                   # Deploy to Together infrastructure
jig status                   # Check deployment status
jig logs                     # View logs

5. Send Requests

Check the health endpoint:

curl https://api.together.ai/v1/deployments/my-inference-service/health \
  -H "Authorization: Bearer $TOGETHER_API_KEY"

Submit a job via the Queue API:

curl -X POST "https://api.together.ai/v1/queue/submit" \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-inference-service",
    "payload": {"prompt": "Hello world"},
    "priority": 1
  }'

Response:

{
  "request_id": "req_abc123",
  "status": "pending"
}

Poll for the result:

curl "https://api.together.ai/v1/queue/status?model=my-inference-service&request_id=req_abc123" \
  -H "Authorization: Bearer $TOGETHER_API_KEY"

Response (when complete):

{
  "request_id": "req_abc123",
  "model": "my-inference-service",
  "status": "done",
  "outputs": {"output": "..."}
}

Or use the Python requests library:

import os
import requests

response = requests.post(
    "https://api.together.ai/v1/queue/submit",
    headers={"Authorization": f"Bearer {os.environ['TOGETHER_API_KEY']}"},
    json={
        "model": "my-inference-service",
        "payload": {"prompt": "Hello world"},
        "priority": 1,
    },
)
print(response.json())

Or submit directly via the Jig CLI:

together beta jig submit --payload '{"prompt": "Hello world"}' --watch

Sprocket SDK

The SDK provides the SprocketWorker base class:

• setup(): Called once at startup — load models, warm up caches
• predict(request): Called per request — process input and return output
• File handling: Upload/download files within predictions
• GPU access: Full CUDA access inside the container

Queue API

For async workloads, use the Queue API for job submission with:

• Priority-based fair queuing
• Progress tracking
• Job status polling

Key Jig CLI Commands

All commands are subcommands of together beta jig. Use --config <path> to specify a custom config file (default: pyproject.toml).

Build and Deploy

Deployment Management

Queue

Secrets

Volumes

Resources

• Full Jig CLI reference: See references/jig-cli.md
• Sprocket SDK reference: See references/sprocket-sdk.md
• App template: See scripts/sprocket_hello_world.py — minimal Sprocket worker with pyproject.toml example
• Official docs: Dedicated Container Inference
• Official docs: Containers Quickstart
• API reference: Deployments API