Large File Uploads and Million-Row Imports in Laravel: S3 Presigned URLs, Queued Batches and Streamed Exports

Almost every Laravel app eventually grows a "bring your data with you" feature: a client uploads a 900 MB video, an operations team drops in a 4-million-row CSV of products, finance exports a year of transactions to Excel. These features are where well-built Laravel apps fall over, because the naive version — $request->file('csv') into a controller, Model::create() in a loop, return Excel::download() — runs inside a single web request with a 60-second timeout and a 512 MB memory limit.

This tutorial builds the production version of both directions: direct-to-S3 uploads that never touch your PHP process, and chunked, queued imports and exports that survive millions of rows. Examples target Laravel 11–13 on PHP 8.2+.

Why the naive version breaks

Four limits bite, usually in this order:

1. upload_max_filesize / post_max_size — PHP rejects the request before your code runs, and the error is invisible to your validator.
2. Request timeout — nginx, PHP-FPM, and load balancers all cap request duration. A 5-minute import is a 504.
3. Memory — file(), Excel::toArray(), and ->get() on a big table all load everything into RAM.
4. Partial failure — row 1.8 million violates a unique constraint, the request dies, and nobody can tell which rows landed.

The fix is the same architectural move in both directions: move bytes out of the request, and move work into queued jobs that report progress and can be retried.

Part 1: direct-to-S3 uploads with presigned URLs

Stop proxying large files through PHP. The browser uploads straight to object storage using a short-lived signed URL, then tells your app the key it wrote.

Install the S3 driver and configure a disk:

composer require league/flysystem-aws-s3-v3 "^3.0"

// config/filesystems.php
's3' => [
    'driver' => 's3',
    'key' => env('AWS_ACCESS_KEY_ID'),
    'secret' => env('AWS_SECRET_ACCESS_KEY'),
    'region' => env('AWS_DEFAULT_REGION'),
    'bucket' => env('AWS_BUCKET'),
    'visibility' => 'private',
    'throw' => true,
],

Issue the signed upload URL from a small endpoint. Constrain everything you can: content type, size, and a key namespaced to the authenticated user.

class UploadUrlController
{
    public function store(Request $request)
    {
        $data = $request->validate([
            'filename' => ['required', 'string', 'max:255'],
            'content_type' => ['required', 'in:video/mp4,text/csv,application/pdf'],
            'size' => ['required', 'integer', 'max:'.(2 * 1024 * 1024 * 1024)],
        ]);

        $key = sprintf(
            'uploads/%s/%s/%s',
            $request->user()->id,
            Str::uuid(),
            Str::of($data['filename'])->basename()->slug()->append('.', pathinfo($data['filename'], PATHINFO_EXTENSION))
        );

        $client = Storage::disk('s3')->getClient();

        $command = $client->getCommand('PutObject', [
            'Bucket' => config('filesystems.disks.s3.bucket'),
            'Key' => $key,
            'ContentType' => $data['content_type'],
        ]);

        return [
            'key' => $key,
            'url' => (string) $client->createPresignedRequest($command, '+15 minutes')->getUri(),
            'headers' => ['Content-Type' => $data['content_type']],
        ];
    }
}

The browser then PUTs the file at that URL and posts the key back:

const { key, url, headers } = await (await fetch('/upload-url', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'X-CSRF-TOKEN': token },
  body: JSON.stringify({ filename: file.name, content_type: file.type, size: file.size }),
})).json()

await fetch(url, { method: 'PUT', headers, body: file })

await fetch('/documents', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'X-CSRF-TOKEN': token },
  body: JSON.stringify({ key }),
})

Two rules that matter in review:

• Never trust the key. Validate on the finalise endpoint that the key starts with uploads/{auth_id}/ and that the object actually exists (Storage::disk('s3')->exists($key)) with a plausible size before you attach it to a model.
• Set a lifecycle rule on the uploads/ prefix to expire orphans after a few days. Abandoned uploads are otherwise permanent storage cost.

If you need uploads above 5 GB, resumability, or progress on flaky mobile connections, use multipart uploads (CreateMultipartUpload plus a signed URL per part) or put Uppy/tus in front of the same S3 bucket. The server-side contract does not change.

For serving files back, generate short-lived signed download URLs rather than streaming through the app:

return Storage::disk('s3')->temporaryUrl($document->key, now()->addMinutes(5), [
    'ResponseContentDisposition' => 'attachment; filename="'.$document->original_name.'"',
]);

Part 2: importing millions of rows without exploding

A CSV import has three phases: ingest the file, split it into chunks, process chunks in parallel. Model the import itself so users can watch it.

Schema::create('imports', function (Blueprint $table) {
    $table->id();
    $table->foreignId('user_id')->constrained();
    $table->string('disk')->default('s3');
    $table->string('path');
    $table->string('status')->default('pending')->index();
    $table->unsignedBigInteger('rows_total')->default(0);
    $table->unsignedBigInteger('rows_processed')->default(0);
    $table->unsignedBigInteger('rows_failed')->default(0);
    $table->uuid('batch_id')->nullable();
    $table->timestamps();
});

Read the file as a stream

Never load the whole CSV. Storage::readStream() plus a generator keeps memory flat regardless of file size:

class CsvReader
{
    public static function rows(string $disk, string $path): \Generator
    {
        $handle = Storage::disk($disk)->readStream($path);
        $header = null;

        try {
            while (($row = fgetcsv($handle)) !== false) {
                if ($header === null) {
                    $header = array_map(fn ($h) => Str::snake(trim($h)), $row);
                    continue;
                }

                if (count($row) !== count($header)) {
                    continue;
                }

                yield array_combine($header, $row);
            }
        } finally {
            fclose($handle);
        }
    }
}

For quoted CSV edge cases, BOM handling, and encoding conversion, league/csv is worth the dependency and reads streams the same way.

Chunk into a queue batch

Use LazyCollection::chunk() to build jobs of a few hundred rows each, then dispatch them as a batch so you get progress, cancellation, and a completion hook for free.

class ProcessImport implements ShouldQueue
{
    public $timeout = 900;

    public function __construct(public Import $import) {}

    public function handle(): void
    {
        $jobs = LazyCollection::make(fn () => yield from CsvReader::rows($this->import->disk, $this->import->path))
            ->chunk(500)
            ->map(fn (LazyCollection $rows) => new ImportRowChunk($this->import->id, $rows->values()->all()));

        $batch = Bus::batch($jobs)
            ->name("import:{$this->import->id}")
            ->allowFailures()
            ->then(fn () => $this->import->update(['status' => 'completed']))
            ->catch(fn () => $this->import->update(['status' => 'failed']))
            ->dispatch();

        $this->import->update(['status' => 'processing', 'batch_id' => $batch->id]);
    }
}

->allowFailures() is deliberate: one bad chunk should not abandon the other 7,000.

Make the row worker idempotent

Chunk jobs get retried. Retries must not duplicate rows, so key every write on something stable from the source data and use upsert() — one statement per chunk instead of 500.

class ImportRowChunk implements ShouldQueue
{
    use Batchable;

    public $tries = 3;
    public $backoff = [10, 60];

    public function __construct(public int $importId, public array $rows) {}

    public function handle(): void
    {
        if ($this->batch()?->cancelled()) {
            return;
        }

        $import = Import::findOrFail($this->importId);
        $valid = [];
        $failed = 0;

        foreach ($this->rows as $row) {
            $validator = Validator::make($row, [
                'sku' => ['required', 'string', 'max:64'],
                'name' => ['required', 'string', 'max:255'],
                'price_cents' => ['required', 'integer', 'min:0'],
            ]);

            if ($validator->fails()) {
                $failed++;
                ImportError::create([
                    'import_id' => $this->importId,
                    'payload' => $row,
                    'errors' => $validator->errors()->toArray(),
                ]);
                continue;
            }

            $valid[] = $validator->validated() + ['updated_at' => now(), 'created_at' => now()];
        }

        if ($valid !== []) {
            Product::upsert($valid, uniqueBy: ['sku'], update: ['name', 'price_cents', 'updated_at']);
        }

        $import->incrementEach([
            'rows_processed' => count($valid),
            'rows_failed' => $failed,
        ]);
    }
}

Notes from real imports:

• Collect errors, do not throw them. An import_errors table that the user can download as a CSV turns a support ticket into self-service.
• Disable what you do not need. For bulk loads, skip model events (Model::withoutEvents()), and turn off query logging (DB::disableQueryLog()) in long-running workers.
• Watch the database, not the queue. Twenty parallel workers doing upsert on a table with six indexes will saturate write IOPS long before PHP struggles. Tune chunk size and worker concurrency together; 500 rows per job and 4–8 workers is a sane starting point.
• Count rows first if you want a real percentage. A cheap streaming pass that only counts newlines is fast and gives rows_total.

Show progress

The batch record already has the numbers, so a status endpoint is trivial and polls cheaply:

public function show(Import $import)
{
    $batch = $import->batch_id ? Bus::findBatch($import->batch_id) : null;

    return [
        'status' => $import->status,
        'processed' => $import->rows_processed,
        'failed' => $import->rows_failed,
        'total' => $import->rows_total,
        'progress' => $batch?->progress() ?? 0,
    ];
}

Wire it to a Livewire wire:poll.3s component, an Inertia 2 polling prop, or a Reverb broadcast — the backend contract is the same.

Part 3: exports that do not run out of memory

Exports fail for the mirror-image reason: building a 200,000-row spreadsheet in memory. Two patterns cover nearly everything.

CSV, streamed row by row. Use a cursor so Eloquent hydrates one model at a time:

return response()->streamDownload(function () {
    $out = fopen('php://output', 'w');
    fputcsv($out, ['id', 'email', 'total_cents', 'created_at']);

    Order::query()
        ->with('customer:id,email')
        ->orderBy('id')
        ->lazyById(1000)
        ->each(function (Order $order) use ($out) {
            fputcsv($out, [$order->id, $order->customer->email, $order->total_cents, $order->created_at->toIso8601String()]);
        });

    fclose($out);
}, 'orders.csv', ['Content-Type' => 'text/csv']);

lazyById() beats chunk() here because it is stable under concurrent inserts and does not use growing OFFSET scans.

Anything large or slow: queue it and email a signed link. A streamed response still holds a web worker open for minutes and dies if the client disconnects.

class ExportOrders implements ShouldQueue
{
    public $timeout = 1800;

    public function __construct(public int $userId, public array $filters) {}

    public function handle(): void
    {
        $path = "exports/{$this->userId}/".Str::uuid().'.csv';
        $temp = tempnam(sys_get_temp_dir(), 'export');
        $out = fopen($temp, 'w');

        fputcsv($out, ['id', 'email', 'total_cents']);

        Order::filter($this->filters)->orderBy('id')->lazyById(1000)
            ->each(fn (Order $o) => fputcsv($out, [$o->id, $o->customer_email, $o->total_cents]));

        fclose($out);

        Storage::disk('s3')->writeStream($path, fopen($temp, 'r'));
        @unlink($temp);

        User::find($this->userId)->notify(new ExportReady($path));
    }
}

The notification links to Storage::disk('s3')->temporaryUrl($path, now()->addHours(24)). Lifecycle-expire the exports/ prefix; nobody wants last February's PII sitting in a bucket forever.

For genuine .xlsx output, maatwebsite/excel with FromQuery plus WithChunkReading (or openspout directly) writes in a streaming fashion. Anything above roughly 100,000 rows is usually better served as CSV or Parquet — Excel itself starts to struggle, and users generally want the data in a database anyway.

Testing this without giant fixtures

it('imports a csv and records bad rows', function () {
    Storage::fake('s3');
    Bus::fake();

    Storage::disk('s3')->put('imports/products.csv', <<<'CSV'
    sku,name,price_cents
    ABC-1,Widget,1999
    ,Broken,100
    CSV);

    $import = Import::factory()->create(['disk' => 's3', 'path' => 'imports/products.csv']);

    (new ProcessImport($import))->handle();

    Bus::assertBatched(fn ($batch) => $batch->jobs->count() === 1);
});

Then test ImportRowChunk directly with a handful of array rows — that is where the business logic lives. Add one test that runs the same chunk twice and asserts the row count did not change; that is your idempotency guarantee, and it is the test that saves you at 2 a.m.

Checklist before you ship

• Large files go browser → object storage, never through PHP.
• Signed URLs are short-lived, content-type constrained, and keys are namespaced per user and re-validated server side.
• Imports are streamed, chunked, batched, idempotent, and retryable.
• Row-level failures are stored and downloadable, not thrown away.
• Exports over a few thousand rows are queued and delivered as an expiring signed link.
• Lifecycle rules expire both uploads/ and exports/.
• Queue workers have timeouts and memory limits set deliberately, and Horizon alerts on a growing wait time.

Get those in place and "we need to import their data" stops being a project risk and becomes a Tuesday.

---

Running an import or file pipeline that times out, duplicates records, or quietly drops rows? Get in touch — we do this work on Laravel projects every week, from a one-week rescue to ongoing platform engineering.