Dribl Crawling

Overview

Extract clubs and fixtures data from https://fv.dribl.com/fixtures/ (SPA with Cloudflare protection) using real browser automation with playwright-core. Two-phase workflow: extraction (raw API data) → transformation (validated, merged data).

Purpose: Crawl dribl.com to maintain up-to-date clubs and fixtures data for Williamstown SC website.

Architecture

Data flow:

dribl API → data/external/fixtures/{team}/ (raw) → transform → data/matches/ (validated)
dribl API → data/external/clubs/ (raw) → transform → data/clubs/ (validated)

Two-phase pattern:

1. Extraction: Playwright intercepts API requests, saves raw JSON
2. Transformation: Read raw data, validate with Zod, transform, deduplicate, save

Key technologies:

• playwright-core (real Chrome browser)
• Zod validation schemas
• TypeScript with tsx runner

Clubs Extraction

Reference: bin/crawlClubs.ts

Pattern:

// Launch browser
const browser = await chromium.launch({
	headless: false,
	channel: 'chrome'
});

// Custom user agent (bypass detection)
const context = await browser.newContext({
	userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36...',
	viewport: { width: 1280, height: 720 }
});

// Intercept API request
const [clubsResponse] = await Promise.all([
	page.waitForResponse((response) => response.url().startsWith(clubsApiUrl) && response.ok(), {
		timeout: 60_000
	}),
	page.goto(url, { waitUntil: 'domcontentloaded' })
]);

// Validate and save
const rawData = await clubsResponse.json();
const validated = externalApiResponseSchema.parse(rawData);
writeFileSync(outputPath, JSON.stringify(validated, null, '\t') + '\n');

API endpoint:

• URL: https://mc-api.dribl.com/api/list/clubs?disable_paging=true
• Response: JSON with data array of club objects
• Validation: externalApiResponseSchema (src/types/matches.ts)

Output:

• Path: data/external/clubs/clubs.json
• Format: Single JSON file with all clubs

CLI args:

• --url <fixtures-page-url> (optional, defaults to standard fixtures page)

Fixtures Extraction

Pattern (implemented in bin/crawlFixtures.ts):

Steps:

1. Navigate to https://fv.dribl.com/fixtures/
2. Wait for SPA to load (waitUntil: 'domcontentloaded')
3. Apply filters (REQUIRED):
   • Season (e.g., “2025”)
   • Competition (e.g., “FFV”)
   • League (e.g., “state-league-2-men-s-north-west”)
4. Intercept /api/fixtures responses
5. Handle pagination:
   • Detect “Load more” button in DOM
   • Click button to load next chunk
   • Wait for new API response
   • Repeat until no more data
6. Save each chunk as chunk-{index}.json

API endpoint:

• URL: https://mc-api.dribl.com/api/fixtures
• Query params: season, competition, league (from filters)
• Response: JSON with data array, links (next/prev), meta (cursors)
• Validation: externalFixturesApiResponseSchema

Output:

• Path: data/external/fixtures/{team}/chunk-0.json, chunk-1.json, etc.
• Format: Multiple JSON files (one per “Load more” click)
• Naming: chunk-{index}.json where index starts at 0

CLI args:

• --team <slug> (required) – Team slug for output folder (e.g., “state-league-2-men-s-north-west”)
• --league <slug> (required) – League slug for filtering (e.g., “State League 2 Men’s – North-West”)
• --season <year> (optional, default to current year)
• --competition <id> (optional, default to FFV)

Clubs Transformation

Reference: bin/syncClubs.ts

Pattern:

// Load external data
const externalResponse = loadExternalData(); // from data/external/clubs/
const validated = externalApiResponseSchema.parse(externalResponse);

// Transform to internal format
const apiClubs = externalResponse.data.map((externalClub) => transformExternalClub(externalClub));

// Load existing clubs
const existingFile = loadExistingClubs(); // from data/clubs/

// Merge (deduplicate by externalId)
const clubsMap = new Map<string, Club>();
for (const club of existingClubs) {
	clubsMap.set(club.externalId, club);
}
for (const apiClub of apiClubs) {
	clubsMap.set(apiClub.externalId, apiClub); // update or add
}

// Sort by name
const mergedClubs = Array.from(clubsMap.values()).sort((a, b) => a.name.localeCompare(b.name));

// Save
writeFileSync(CLUBS_FILE_PATH, JSON.stringify({ clubs: mergedClubs }, null, '\t'));

Transform service: src/lib/clubService.ts

• transformExternalClub(): Converts external club format to internal format
• Maps fields: id→externalId, attributes.name→name/displayName, etc.
• Normalizes address (combines address_line_1 + address_line_2)
• Maps socials array (name→platform, value→url)
• Validates output with clubSchema

Output:

• Path: data/clubs/clubs.json
• Format: { clubs: Club[] }

Fixtures Transformation

Reference: bin/syncFixtures.ts

Pattern:

// Read all chunk files
const teamDir = path.join(EXTERNAL_DIR, team);
const files = await fs.readdir(teamDir);
const chunkFiles = files.filter((f) => f.match(/^chunk-\d+\.json$/)).sort(); // natural number sort

// Load and validate each chunk
const responses: ExternalFixturesApiResponse[] = [];
for (const file of chunkFiles) {
	const content = await fs.readFile(path.join(teamDir, file), 'utf-8');
	const validated = externalFixturesApiResponseSchema.parse(JSON.parse(content));
	responses.push(validated);
}

// Transform all fixtures
const allFixtures = [];
for (const response of responses) {
	for (const externalFixture of response.data) {
		const fixture = transformExternalFixture(externalFixture);
		allFixtures.push(fixture);
	}
}

// Deduplicate (by round + homeTeamId + awayTeamId)
const seen = new Set<string>();
const deduplicated = allFixtures.filter((f) => {
	const key = `${f.round}-${f.homeTeamId}-${f.awayTeamId}`;
	if (seen.has(key)) return false;
	seen.add(key);
	return true;
});

// Sort by round, then date
const sorted = deduplicated.sort((a, b) => {
	if (a.round !== b.round) return a.round - b.round;
	return a.date.localeCompare(b.date);
});

// Calculate metadata
const totalRounds = Math.max(...sorted.map((f) => f.round), 0);

// Save
const fixtureData = {
	competition: 'FFV',
	season: 2025,
	totalFixtures: sorted.length,
	totalRounds,
	fixtures: sorted
};
writeFileSync(outputPath, JSON.stringify(fixtureData, null, '\t'));

Transform service: src/lib/matches/fixtureTransformService.ts

• transformExternalFixture(): Converts external fixture format to internal format
• Parses round number (e.g., “R1” → 1)
• Formats date/time/day strings (ISO date, 24h time, weekday name)
• Combines ground + field names for address
• Finds club external IDs by matching team names/logos
• Validates output with fixtureSchema

Output:

• Path: data/matches/{team}.json
• Format: { competition, season, totalFixtures, totalRounds, fixtures: Fixture[] }

CLI args:

• --team <slug> (required) – Team slug to sync (e.g., “state-league-2-men-s-north-west”)

Validation Schemas

Reference: src/types/matches.ts

External schemas (API responses):

• externalApiResponseSchema: Clubs API response
• externalClubSchema: Single club object
• externalFixturesApiResponseSchema: Fixtures API response
• externalFixtureSchema: Single fixture object

Internal schemas (transformed data):

• clubSchema: Single club
• clubsSchema: Clubs file ({ clubs: Club[] })
• fixtureSchema: Single fixture
• fixtureDataSchema: Fixtures file ({ competition, season, totalFixtures, totalRounds, fixtures })

Pattern: Always validate at boundaries (API → external schema, transform → internal schema)

CI Integration

Reference: .github/workflows/crawl-clubs.yml

Linux setup (GitHub Actions):

- name: Install Chrome
  run: npx playwright install --with-deps chrome

- name: Crawl clubs
  run: npm run crawl:clubs:ci -- ${{ inputs.url && format('--url "{0}"', inputs.url) || '' }}

Key points:

• Use xvfb-run prefix on Linux for headless Chrome (e.g., xvfb-run npm run crawl:clubs)
• Install with --with-deps flag to get system dependencies
• Set appropriate timeout (5 min for clubs, may need more for fixtures)
• Upload artifacts for data files

Package.json scripts pattern:

{
	"crawl:clubs": "tsx bin/crawlClubs.ts",
	"crawl:clubs:ci": "xvfb-run tsx bin/crawlClubs.ts",
	"sync:clubs": "tsx bin/syncClubs.ts",
	"sync:fixtures": "tsx bin/syncFixtures.ts"
}

Best Practices

Logging:

• Use emoji logging for clarity:
  • ✓ / ✅ – Success
  • ❌ – Error
  • 📂 – File operations
  • 🔄 – Processing/transformation
• Log counts and progress for large operations

Error handling:

• Try/catch at top level
• Special handling for ZodError (print issues)
• Exit with code 1 on failure
• Close browser in finally block

File operations:

• Always use mkdirSync(path, { recursive: true }) before writing
• Format JSON with tabs: JSON.stringify(data, null, '\t')
• Add newline at end of file: content + '\n'
• Use absolute paths with resolve(__dirname, '../relative/path')

Data separation:

• Keep raw external data in data/external/ (gitignored)
• Keep transformed data in data/ (committed)
• Never commit external API responses directly

Validation:

• Validate immediately after receiving API data
• Validate before writing transformed data
• Use descriptive error messages with file paths

CLI arguments:

• Use Commander library for consistent CLI parsing
• Define options with .option() or .requiredOption()
• Provide defaults for optional args
• Commander auto-generates help text and validates required args

Common Patterns

Reading chunks:

const files = await fs.readdir(dir);
const chunks = files
	.filter((f) => f.match(/^chunk-\d+\.json$/))
	.sort((a, b) => {
		const numA = parseInt(a.match(/\d+/)?.[0] || '0', 10);
		const numB = parseInt(b.match(/\d+/)?.[0] || '0', 10);
		return numA - numB;
	});

Deduplication:

const seen = new Set<string>();
const unique = items.filter((item) => {
	const key = computeKey(item);
	if (seen.has(key)) return false;
	seen.add(key);
	return true;
});

Merge with existing:

const map = new Map<string, T>();
existing.forEach((item) => map.set(item.id, item));
incoming.forEach((item) => map.set(item.id, item)); // update or add
const merged = Array.from(map.values());

Browser cleanup:

let browser: Browser | undefined;
try {
  browser = await chromium.launch(...);
  // work
} finally {
  if (browser) await browser.close();
}