Designing a Media Processing Job System on Cloudflare Pages, D1, R2 and KV
Architecture notes for signed uploads, job state, polling, SSE progress, retention and webhooks.
Bottom line: A media job needs three things: durable state, temporary object storage, and a way to report progress. On Cloudflare, a natural design uses D1 for users, jobs, files, credit transactions and payment records; R2 for private media objects; and KV for short-lived status caches, idempotency hints and rate-limit counters.
Direct upload keeps the Worker outside the large-file data path. The client asks for an upload policy, receives a signed destination or short-lived R2 credentials, uploads bytes directly, then creates a job referencing the object. This avoids buffering video through the Worker and improves both latency and reliability.
Job state should be explicit: created, awaiting_upload, queued, processing, succeeded, failed, expired and deleted. The client can poll a status endpoint for compatibility and subscribe to SSE for responsive progress. SSE is simpler than WebSocket for one-way status updates and works well behind HTTP infrastructure.
Credits require transaction discipline. Before allowing a paid AI job, reserve credits with a ledger row and a job identifier. If the job succeeds, settle the reservation; if it fails for an infrastructure reason, release it. Idempotency keys prevent duplicate charges from retries. Payment webhooks should be stored before subscription state is changed.
R2 lifecycle rules should enforce retention independently of application cleanup. Free objects can expire after 24 hours and Pro objects after seven days, while job records may retain minimal metadata longer for billing and abuse prevention. The UI must state these windows before upload.
A safe MVP can use deterministic mock AI output when provider secrets are absent. This lets engineering test uploads, queue state, polling, downloads, consent and billing plumbing without pretending that a real model ran. When ASR, TTS or LLM credentials are configured, the same job pipeline swaps in the real provider.