How to Migrate from S3 Direct Links to a CDN

Back to Blog
Lukas Gisder-Dubé
•
•
7 min read
cdnawss3migrationtutorial

How to Migrate from S3 Direct Links to a CDN

A few months ago, I inherited a codebase where every image URL looked like this:

text
https://my-bucket.s3.us-east-1.amazonaws.com/uploads/product-123.jpg

The app worked, but users in Asia were complaining about slow image loads. And the AWS bill had a surprising line item for S3 data transfer that grew every month.

The fix was straightforward: put a CDN in front of those assets. But migrating hundreds of existing URLs without breaking anything? That required a plan.

Here's the step-by-step process I used, and how you can do the same.

Why Migrate Away from Direct S3 Links?

Direct S3 URLs have several drawbacks:

Performance. S3 serves files from a single region. Users far from that region experience latency. A CDN caches content at 400+ edge locations worldwide.

Cost. S3 data transfer starts at $0.09/GB for the first 10TB in most regions. CDN pricing can be more competitive depending on the provider and scale—always compare based on your specific usage patterns.

Features. S3 is storage. CDNs add caching headers, image optimization, and better analytics.

Flexibility. Direct S3 URLs tie you to AWS. A CDN URL gives you freedom to change storage backends later.

The Migration Strategy

There are two approaches:

Option A: CDN in front of S3. Keep files in S3, route requests through CDN. Quick to set up, but you're still paying for S3 storage.

Option B: Full migration. Move files to the CDN's storage. Cleaner long-term, but requires updating all URLs.

I'll show you Option B since it gives you the most benefits and simplest architecture.

Step 1: Inventory Your Assets

First, understand what you're migrating. List all files in your S3 bucket:

bash
aws s3 ls s3://my-bucket/uploads/ --recursive > inventory.txt

Check the total count and size:

bash
wc -l inventory.txt  # Number of files
aws s3 ls s3://my-bucket/uploads/ --recursive --summarize | tail -2

For large buckets (10,000+ files), consider migrating in batches by folder or date.

Step 2: Set Up Your CDN

Create an account on your CDN provider and get your API credentials. With easyCDN, grab your secret key from the dashboard.

Set up environment variables:

bash
# .env
EASYCDN_SECRET_KEY=your_secret_key
AWS_ACCESS_KEY_ID=your_aws_key
AWS_SECRET_ACCESS_KEY=your_aws_secret
AWS_REGION=us-east-1
S3_BUCKET=my-bucket

Step 3: Build the Migration Script

Here's a script that downloads from S3 and uploads to the CDN. First, install the required dependencies:

bash
npm install @aws-sdk/client-s3 @easycdn/server

Now create the migration script:

ts
import { S3Client, GetObjectCommand, ListObjectsV2Command } from '@aws-sdk/client-s3'
import { createClient } from '@easycdn/server'

const s3 = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
})

const cdn = createClient({
  secretKey: process.env.EASYCDN_SECRET_KEY!,
})

interface MigrationResult {
  s3Key: string
  cdnUrl: string
  success: boolean
  error?: string
}

async function streamToBuffer(stream: NodeJS.ReadableStream): Promise<Buffer> {
  const chunks: Buffer[] = []
  for await (const chunk of stream) {
    chunks.push(Buffer.from(chunk))
  }
  return Buffer.concat(chunks)
}

async function migrateFile(key: string): Promise<MigrationResult> {
  try {
    // Download from S3
    const response = await s3.send(
      new GetObjectCommand({
        Bucket: process.env.S3_BUCKET,
        Key: key,
      })
    )

    const buffer = await streamToBuffer(response.Body as NodeJS.ReadableStream)

    // Upload to CDN
    const result = await cdn.upload(buffer, {
      fileName: key,
    })

    return {
      s3Key: key,
      cdnUrl: result.asset.url,
      success: true,
    }
  } catch (error) {
    return {
      s3Key: key,
      cdnUrl: '',
      success: false,
      error: error instanceof Error ? error.message : 'Unknown error',
    }
  }
}

async function migrateFolder(prefix: string) {
  const results: MigrationResult[] = []

  let continuationToken: string | undefined

  do {
    const listResponse = await s3.send(
      new ListObjectsV2Command({
        Bucket: process.env.S3_BUCKET,
        Prefix: prefix,
        ContinuationToken: continuationToken,
      })
    )

    for (const object of listResponse.Contents || []) {
      if (!object.Key) continue

      console.log(`Migrating: ${object.Key}`)
      const result = await migrateFile(object.Key)
      results.push(result)

      if (!result.success) {
        console.error(`Failed: ${object.Key} - ${result.error}`)
      }
    }

    continuationToken = listResponse.NextContinuationToken
  } while (continuationToken)

  return results
}

// Run migration
migrateFolder('uploads/').then((results) => {
  const successful = results.filter((r) => r.success)
  const failed = results.filter((r) => !r.success)

  console.log(`\nMigration complete:`)
  console.log(`  Successful: ${successful.length}`)
  console.log(`  Failed: ${failed.length}`)

  // Save URL mapping for database updates
  const urlMap = successful.map((r) => ({
    old: `https://${process.env.S3_BUCKET}.s3.${process.env.AWS_REGION}.amazonaws.com/${r.s3Key}`,
    new: r.cdnUrl,
  }))

  console.log('\nURL mapping:', JSON.stringify(urlMap, null, 2))
})

Step 4: Update Database References

Now you need to update all the URLs stored in your database. Since the CDN generates unique URLs for each asset, you can't do a simple find-and-replace on the base URL. Use the URL mapping generated by the migration script in Step 3:

ts
import { db } from './database'

async function updateUrls(urlMap: { old: string; new: string }[]) {
  for (const { old: oldUrl, new: newUrl } of urlMap) {
    await db.query(
      'UPDATE products SET image_url = $1 WHERE image_url = $2',
      [newUrl, oldUrl]
    )
  }
}

Save the URL mapping to a file before running updates, so you have a record for rollback if needed:

ts
import { writeFile } from 'fs/promises'

// After migration in Step 3
await writeFile('url-mapping.json', JSON.stringify(urlMap, null, 2))

Step 5: Update Application Code

Search your codebase for S3 URL patterns:

bash
grep -r "s3.amazonaws.com" --include="*.ts" --include="*.tsx" ./src

Replace hardcoded S3 references with CDN URLs or environment variables:

ts
// Before
const imageUrl = `https://my-bucket.s3.us-east-1.amazonaws.com/${key}`

// After
const imageUrl = `${process.env.CDN_URL}/${key}`

Step 6: Redirect Old URLs (Optional)

If external sites link to your S3 URLs, set up redirects to avoid broken links. Load the URL mapping from the migration and use it for lookups:

ts
import urlMapping from './url-mapping.json'

// Build a lookup from the old S3 path to the new CDN URL
const redirectMap = new Map(
  urlMapping.map(({ old, new: cdnUrl }: { old: string; new: string }) => {
    const path = new URL(old).pathname
    return [path, cdnUrl]
  })
)

app.use((req, res, next) => {
  const cdnUrl = redirectMap.get(req.path)
  if (cdnUrl) {
    return res.redirect(301, cdnUrl)
  }
  next()
})

Or configure S3 website hosting with redirect rules if you want S3 to handle the redirects directly.

Step 7: Verify and Clean Up

After migration, verify everything works:

  1. Spot-check random URLs in the browser
  2. Run your test suite
  3. Check application logs for 404s
  4. Monitor CDN analytics for traffic

Once confident, you can:

  • Delete files from S3 (or move to cheaper storage class)
  • Remove S3 read permissions from your application
  • Update documentation

Handling Large Migrations

For buckets with millions of files:

Parallelize uploads. Process multiple files concurrently:

ts
import pLimit from 'p-limit'

const limit = pLimit(10) // 10 concurrent uploads

const promises = files.map((file) =>
  limit(() => migrateFile(file))
)

const results = await Promise.all(promises)

Migrate incrementally. Move one folder at a time over several days.

Use a job queue. For very large migrations, use a queue like BullMQ to handle retries and monitoring.

Wrapping Up

Migrating from S3 direct links to a CDN is mostly a matter of patience and careful URL updates. The technical work is straightforward—download, upload, replace URLs.

The payoff is immediate: faster load times for global users, potentially lower costs, and a cleaner architecture.

If you're looking for a smooth migration path, easyCDN makes it simple to upload your existing assets and get CDN URLs in return. The free tier gives you room to test the migration before committing, and the SDK handles all the upload complexity.

I've migrated dozens of projects this way, and the speed boost is noticeable immediately. Your users won't know you changed anything—they'll just notice that images load faster.

Previous Post

Why Your Web Vitals Are Suffering (And How to Fix Them)

Diagnosing Core Web Vitals issues related to asset delivery

Next Post

How CDNs Protect Your Site from DDoS and Security Threats

Understanding CDN security features and best practices for protection

Ready to host your assets?

Sign up, connect your bucket, and start serving your assets.

14-day free trial • From $9/month • Bring your own bucket