Transformers.js – Machine Learning for JavaScript

Transformers.js enables running state-of-the-art machine learning models directly in JavaScript, both in browsers and Node.js environments, with no server required.

When to Use This Skill

Use this skill when you need to:

• Run ML models for text analysis, generation, or translation in JavaScript
• Perform image classification, object detection, or segmentation
• Implement speech recognition or audio processing
• Build multimodal AI applications (text-to-image, image-to-text, etc.)
• Run models client-side in the browser without a backend

Installation

NPM Installation

npm install @huggingface/transformers

Browser Usage (CDN)

<script type="module">
  import { pipeline } from 'https://cdn.jsdelivr.net/npm/@huggingface/transformers';
</script>

Core Concepts

1. Pipeline API

The pipeline API is the easiest way to use models. It groups together preprocessing, model inference, and postprocessing:

import { pipeline } from '@huggingface/transformers';

// Create a pipeline for a specific task
const classifier = await pipeline('sentiment-analysis');

// Use the pipeline
const result = await classifier('I love transformers!');
// Output: [{ label: 'POSITIVE', score: 0.999817686 }]

// IMPORTANT: Always dispose when done to free memory
await classifier.dispose();

⚠️ Memory Management: All pipelines must be disposed with pipe.dispose() when finished to prevent memory leaks. See Memory Management section below.

2. Model Selection

You can specify a custom model as the second argument:

const pipe = await pipeline(
  'sentiment-analysis',
  'Xenova/bert-base-multilingual-uncased-sentiment'
);

3. Device Selection

Choose where to run the model:

// Run on CPU (default for WASM)
const pipe = await pipeline('sentiment-analysis', 'model-id');

// Run on GPU (WebGPU - experimental)
const pipe = await pipeline('sentiment-analysis', 'model-id', {
  device: 'webgpu',
});

4. Quantization Options

Control model precision vs. performance:

// Use quantized model (faster, smaller)
const pipe = await pipeline('sentiment-analysis', 'model-id', {
  dtype: 'q4',  // Options: 'fp32', 'fp16', 'q8', 'q4'
});

Supported Tasks

Note: All examples below show basic usage. Remember to call pipe.dispose() when you’re finished with any pipeline to free memory. See Memory Management for details.

Natural Language Processing

Text Classification

const classifier = await pipeline('text-classification');
const result = await classifier('This movie was amazing!');

Named Entity Recognition (NER)

const ner = await pipeline('token-classification');
const entities = await ner('My name is John and I live in New York.');

Question Answering

const qa = await pipeline('question-answering');
const answer = await qa({
  question: 'What is the capital of France?',
  context: 'Paris is the capital and largest city of France.'
});

Text Generation

const generator = await pipeline('text-generation', 'onnx-community/gemma-3-270m-it-ONNX');
const text = await generator('Once upon a time', {
  max_new_tokens: 100,
  temperature: 0.7
});

For streaming and chat: See Text Generation Guide for:

• Streaming token-by-token output with TextStreamer
• Chat/conversation format with system/user/assistant roles
• Generation parameters (temperature, top_k, top_p)
• Browser and Node.js examples
• React components and API endpoints

Translation

const translator = await pipeline('translation', 'Xenova/nllb-200-distilled-600M');
const output = await translator('Hello, how are you?', {
  src_lang: 'eng_Latn',
  tgt_lang: 'fra_Latn'
});

Summarization

const summarizer = await pipeline('summarization');
const summary = await summarizer(longText, {
  max_length: 100,
  min_length: 30
});

Zero-Shot Classification

const classifier = await pipeline('zero-shot-classification');
const result = await classifier('This is a story about sports.', ['politics', 'sports', 'technology']);

Computer Vision

Image Classification

const classifier = await pipeline('image-classification');
const result = await classifier('https://example.com/image.jpg');
// Or with local file
const result = await classifier(imageUrl);

Object Detection

const detector = await pipeline('object-detection');
const objects = await detector('https://example.com/image.jpg');
// Returns: [{ label: 'person', score: 0.95, box: { xmin, ymin, xmax, ymax } }, ...]

Image Segmentation

const segmenter = await pipeline('image-segmentation');
const segments = await segmenter('https://example.com/image.jpg');

Depth Estimation

const depthEstimator = await pipeline('depth-estimation');
const depth = await depthEstimator('https://example.com/image.jpg');

Zero-Shot Image Classification

const classifier = await pipeline('zero-shot-image-classification');
const result = await classifier('image.jpg', ['cat', 'dog', 'bird']);

Audio Processing

Automatic Speech Recognition

const transcriber = await pipeline('automatic-speech-recognition');
const result = await transcriber('audio.wav');
// Returns: { text: 'transcribed text here' }

Audio Classification

const classifier = await pipeline('audio-classification');
const result = await classifier('audio.wav');

Text-to-Speech

const synthesizer = await pipeline('text-to-speech', 'Xenova/speecht5_tts');
const audio = await synthesizer('Hello, this is a test.', {
  speaker_embeddings: speakerEmbeddings
});

Multimodal

Image-to-Text (Image Captioning)

const captioner = await pipeline('image-to-text');
const caption = await captioner('image.jpg');

Document Question Answering

const docQA = await pipeline('document-question-answering');
const answer = await docQA('document-image.jpg', 'What is the total amount?');

Zero-Shot Object Detection

const detector = await pipeline('zero-shot-object-detection');
const objects = await detector('image.jpg', ['person', 'car', 'tree']);

Feature Extraction (Embeddings)

const extractor = await pipeline('feature-extraction');
const embeddings = await extractor('This is a sentence to embed.');
// Returns: tensor of shape [1, sequence_length, hidden_size]

// For sentence embeddings (mean pooling)
const extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
const embeddings = await extractor('Text to embed', { pooling: 'mean', normalize: true });

Advanced Configuration

Environment Configuration (env)

The env object provides comprehensive control over Transformers.js execution, caching, and model loading.

Quick Overview:

import { env } from '@huggingface/transformers';

// View version
console.log(env.version); // e.g., '3.8.1'

// Common settings
env.allowRemoteModels = true;  // Load from Hugging Face Hub
env.allowLocalModels = false;  // Load from file system
env.localModelPath = '/models/'; // Local model directory
env.useFSCache = true;         // Cache models on disk (Node.js)
env.useBrowserCache = true;    // Cache models in browser
env.cacheDir = './.cache';     // Cache directory location

Configuration Patterns:

// Development: Fast iteration with remote models
env.allowRemoteModels = true;
env.useFSCache = true;

// Production: Local models only
env.allowRemoteModels = false;
env.allowLocalModels = true;
env.localModelPath = '/app/models/';

// Custom CDN
env.remoteHost = 'https://cdn.example.com/models';

// Disable caching (testing)
env.useFSCache = false;
env.useBrowserCache = false;

For complete documentation on all configuration options, caching strategies, cache management, pre-downloading models, and more, see:

→ Configuration Reference

Working with Tensors

import { AutoTokenizer, AutoModel } from '@huggingface/transformers';

// Load tokenizer and model separately for more control
const tokenizer = await AutoTokenizer.from_pretrained('bert-base-uncased');
const model = await AutoModel.from_pretrained('bert-base-uncased');

// Tokenize input
const inputs = await tokenizer('Hello world!');

// Run model
const outputs = await model(inputs);

Batch Processing

const classifier = await pipeline('sentiment-analysis');

// Process multiple texts
const results = await classifier([
  'I love this!',
  'This is terrible.',
  'It was okay.'
]);

Browser-Specific Considerations

WebGPU Usage

WebGPU provides GPU acceleration in browsers:

const pipe = await pipeline('text-generation', 'Xenova/gpt2', {
  device: 'webgpu',
  dtype: 'fp32'
});

Note: WebGPU is experimental. Check browser compatibility and file issues if problems occur.

WASM Performance

Default browser execution uses WASM:

// Optimized for browsers with quantization
const pipe = await pipeline('sentiment-analysis', 'model-id', {
  dtype: 'q8'  // or 'q4' for even smaller size
});

Progress Tracking & Loading Indicators

Models can be large (ranging from a few MB to several GB). Track download progress by passing a callback to the pipeline() function:

import { pipeline } from '@huggingface/transformers';

// Define progress callback
function onProgress(info) {
  console.log(`${info.status}: ${info.file}`);
  
  if (info.status === 'progress') {
    console.log(`Progress: ${info.progress.toFixed(1)}%`);
  }
}

// Pass callback to pipeline
const classifier = await pipeline('sentiment-analysis', null, {
  progress_callback: onProgress
});

Progress Info Properties:

interface ProgressInfo {
  status: 'initiate' | 'download' | 'progress' | 'done' | 'ready';
  name: string;      // Model id or path
  file: string;      // File being processed
  progress?: number; // Percentage (0-100, only for 'progress' status)
  loaded?: number;   // Bytes downloaded (only for 'progress' status)
  total?: number;    // Total bytes (only for 'progress' status)
}

For complete examples including browser UIs, React components, CLI progress bars, and retry logic, see:

→ Pipeline Options – Progress Callback

Error Handling

try {
  const pipe = await pipeline('sentiment-analysis', 'model-id');
  const result = await pipe('text to analyze');
} catch (error) {
  if (error.message.includes('fetch')) {
    console.error('Model download failed. Check internet connection.');
  } else if (error.message.includes('ONNX')) {
    console.error('Model execution failed. Check model compatibility.');
  } else {
    console.error('Unknown error:', error);
  }
}

Performance Tips

1. Reuse Pipelines: Create pipeline once, reuse for multiple inferences
2. Use Quantization: Start with q8 or q4 for faster inference
3. Batch Processing: Process multiple inputs together when possible
4. Cache Models: Models are cached automatically in ~/.cache/huggingface/
5. WebGPU for Large Models: Use WebGPU for models that benefit from GPU acceleration
6. Prune Context: For text generation, limit max_new_tokens to avoid memory issues
7. Clean Up Resources: Call pipe.dispose() when done to free memory

Memory Management

IMPORTANT: All pipelines (sentiment analysis, image classification, speech recognition, text generation, etc.) must be disposed when you’re done to prevent memory leaks.

Basic Cleanup

import { pipeline } from '@huggingface/transformers';

// Text classification
const classifier = await pipeline('sentiment-analysis');
const result = await classifier('This is great!');
await classifier.dispose();  // ✓ Always dispose when done

// Image classification
const imageClassifier = await pipeline('image-classification');
const imageResult = await imageClassifier('image.jpg');
await imageClassifier.dispose();  // ✓ Free GPU/CPU memory

// Speech recognition
const transcriber = await pipeline('automatic-speech-recognition');
const transcript = await transcriber('audio.wav');
await transcriber.dispose();  // ✓ Clean up audio model

// Any task - same pattern
const anyPipeline = await pipeline('any-task');
// ... use pipeline ...
await anyPipeline.dispose();  // ✓ Always dispose

When to Dispose

✓ Always dispose in these scenarios:

• Application shutdown – Clean up before process exits
• Component unmount – React/Vue/Angular/etc.
• Model switching – Dispose old model before loading new one
• Batch processing complete – After processing a batch in long-running apps
• User logout/session end – Clean up user-specific models
• Memory pressure – When running low on memory

✗ Don’t dispose if:

• You’ll immediately reuse the same model
• The application is terminating anyway

Why Dispose Matters

• Memory leaks – Models consume significant memory (100MB – several GB)
• GPU resources – WebGPU contexts must be released
• CPU resources – WASM instances need cleanup
• Browser limits – Browsers have memory quotas
• Server stability – Long-running servers accumulate memory without cleanup

For more cleanup patterns (worker threads, model switching, monitoring), see:

• Code Examples – Memory Management
• Text Generation Guide – Memory Management

Troubleshooting

Model Not Found

• Verify model exists on Hugging Face Hub
• Check model name spelling
• Ensure model has ONNX files (look for onnx folder in model repo)

Memory Issues

• Use smaller models or quantized versions (dtype: 'q4')
• Reduce batch size
• Limit sequence length with max_length

WebGPU Errors

• Check browser compatibility (Chrome 113+, Edge 113+)
• Try dtype: 'fp16' if fp32 fails
• Fall back to WASM if WebGPU unavailable

Reference Documentation

This Skill

• Pipeline Options – Configure pipeline() with progress_callback, device, dtype, etc.
• Configuration Reference – Global env configuration for caching and model loading
• Text Generation Guide – Streaming, chat format, and generation parameters
• Model Architectures – Supported models and selection tips
• Code Examples – Real-world implementations for different runtimes

Official Transformers.js

• Official docs: https://huggingface.co/docs/transformers.js
• API reference: https://huggingface.co/docs/transformers.js/api/pipelines
• Model hub: https://huggingface.co/models?library=transformers.js
• GitHub: https://github.com/huggingface/transformers.js
• Examples: https://github.com/huggingface/transformers.js/tree/main/examples

Best Practices

1. Always Dispose Pipelines: Call pipe.dispose() when done – critical for preventing memory leaks (see Memory Management)
2. Start with Pipelines: Use the pipeline API unless you need fine-grained control
3. Test Locally First: Test models with small inputs before deploying
4. Monitor Model Sizes: Be aware of model download sizes for web applications
5. Handle Loading States: Show progress indicators for better UX
6. Version Pin: Pin specific model versions for production stability
7. Error Boundaries: Always wrap pipeline calls in try-catch blocks
8. Progressive Enhancement: Provide fallbacks for unsupported browsers
9. Reuse Models: Load once, use many times – don’t recreate pipelines unnecessarily
10. Graceful Shutdown: Dispose models on SIGTERM/SIGINT in servers

Quick Reference: Task IDs

This skill enables you to integrate state-of-the-art machine learning capabilities directly into JavaScript applications without requiring separate ML servers or Python environments.