How to Download AWS S3 Presigned URLs Using curl on Linux

Accessing private objects in Amazon S3 without exposing your AWS credentials is a common requirement for developers and system administrators. Presigned URLs let you grant time-limited access to S3 objects, and curl on Linux makes downloading via HTTP straightforward. In this guide, you’ll learn:

• What S3 presigned URLs are and when to use them
• How to generate presigned URLs with AWS CLI and Python
• Multiple curl techniques for reliable downloads (including -L -J -O)
• Batch downloading, scripting, and automation
• Security best practices and common troubleshooting tips

What Are AWS S3 Presigned URLs?

A presigned URL is a URL that grants temporary access to a specific S3 object. When you create one, AWS:

• Signs the request with your credentials
• Embeds an expiration time
• Allows anyone possessing that URL to perform the specified operation (GET or PUT)

With presigned URLs, you avoid sharing long-lived AWS keys or giving users direct IAM permissions.

Why Use Presigned URLs?

• Limit exposure: No need to embed AWS access keys in client code.
• Time-bound access: URLs expire automatically.
• Universal compatibility: Works with any HTTP client (browser, curl, wget, etc.).
• CI/CD integration: Ideal for uploads (PUT) and downloads (GET) in automated pipelines.

Prerequisites

• AWS CLI (v2 recommended) or boto3 if using Python
• curl installed (sudo apt install curl / yum install curl)
• jq (optional, for parsing JSON in scripts)
• Network connectivity to the S3 bucket’s region endpoint

Generating Presigned URLs

Before you can download, you must first generate a valid presigned URL. Here are two common methods:

AWS CLI Method

1. Configure AWS CLI

   aws configure
   # Enter AWS Access Key, Secret Key, region, output format
   

2. Generate a GET presigned URL

   aws s3 presign s3://my-bucket/path/to/object.txt \
     --expires-in 3600
   

   --expires-in specifies validity in seconds (max 7 days).

Python (boto3) Method

1. Install boto3

   pip install boto3

2. Generate URL in a script

   import boto3
   from botocore.exceptions import ClientError
   
   s3 = boto3.client('s3')
   try:
       url = s3.generate_presigned_url(
           ClientMethod='get_object',
           Params={'Bucket': 'my-bucket', 'Key': 'path/to/object.txt'},
           ExpiresIn=3600
       )
       print(url)
   except ClientError as e:
       print(f"Error generating presigned URL: {e}")
   

3. Run the script

   python generate_presigned.py

Downloading with curl on Linux

Once you have your presigned URL, curl makes the download trivial. Below are several useful patterns.

Basic Download

Use the -O flag to save using the remote filename:

curl -O "https://my-bucket.s3.amazonaws.com/path/to/object.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=…"

• -O writes to object.txt (the basename of the URL).
• Always quote the URL to avoid shell interpretation of & characters.

Saving to a Custom Filename

To specify your own output name:

curl -L -o myfile.txt \
  "https://…presigned-url…"

• -L follows redirects (some buckets issue a 302).
• -o myfile.txt writes to myfile.txt.

Honoring Server-Provided Filenames with -J

If your object was uploaded with a Content-Disposition: attachment; filename=… header, you can have curl honor that suggested filename:

curl -L -J -O "https://my-bucket.s3.amazonaws.com/path/to/object.txt?…"

• -J tells curl to use the filename from the Content-Disposition header.
• Requires -O so that the server-provided name is used instead of the URL basename.

Handling Redirects & Errors

Combine options for maximum robustness:

curl --fail --show-error --location \
     --write-out "\nHTTP %{http_code}\n" \
     --output download.bin \
     "https://…presigned-url…"

• --fail exits non-zero on HTTP errors (4xx/5xx).
• --show-error prints the server’s error response.
• --location is an alias for -L, following redirects.
• --write-out "\nHTTP %{http_code}\n" prints the HTTP status code after the transfer.

Batch Downloading Multiple Presigned URLs

When you have dozens or hundreds of URLs, manual downloads don’t scale. Here are two ways to batch them:

Using a Bash Loop

1. Create urls.txt (one URL per line):

   https://…url1…
   https://…url2…
   

2. Run the loop:

   while IFS= read -r url; do
     curl --fail -O "$url"
   done < urls.txt

Leveraging GNU Parallel

1. Install GNU Parallel:

   sudo apt install parallel

2. Download with a progress bar:

   parallel --bar curl --fail -O {} :::: urls.txt

3. Control concurrency (e.g., 10 at a time):

   parallel -j 10 curl --fail -O {} :::: urls.txt

Automating Downloads in Scripts

Integrate curl into a Bash script for repeatable workflows:

#!/usr/bin/env bash
set -euo pipefail

URL_FILE="urls.txt"
OUTPUT_DIR="downloads"
mkdir -p "$OUTPUT_DIR"

while IFS= read -r url; do
  filename=$(basename "${url%%\?*}")
  echo "Downloading $filename..."
  curl --fail --silent --show-error \
       --location --output "$OUTPUT_DIR/$filename" \
       "$url"
done < "$URL_FILE"

echo "All downloads complete!"

• Strips query parameters to derive filename.
• Saves all files into a downloads/ directory.
• Can be extended with retries, logging, or notifications.

Security Best Practices

• Keep expirations short: Use minutes rather than hours whenever possible.
• Least privilege: Presign only the specific object and operation you need.
• Always use HTTPS: Prevents URL sniffing or tampering.
• Rotate IAM credentials: Limits blast radius if keys leak.
• Audit usage: Enable S3 Access Logs or CloudTrail for monitoring.

Troubleshooting Common Errors

Conclusion & Next Steps

Downloading AWS S3 presigned URLs with curl on Linux is a secure, flexible, and automatable way to access private S3 objects without exposing your AWS credentials. By mastering these techniques—especially the handy -L -J -O pattern—you can integrate presigned‐URL downloads into scripts, CI/CD pipelines, and one-off tasks alike.

Next steps:

• Try generating PUT presigned URLs for secure uploads.
• Build a simple CLI wrapper around curl for repeatable workflows.
• Integrate downloads into your CI/CD tool (Jenkins, GitLab CI, GitHub Actions).
• Set up monitoring and alerts on S3 access logs to audit usage.

FAQ

Can I download multiple objects with one presigned URL?

No. Each presigned URL is valid for a single object and a single operation.

What’s the maximum expiration for a presigned URL?

You can set up to 7 days (604,800 seconds) via AWS SDKs or CLI.

Can I presign a folder or prefix?

No. You must generate an individual URL for each object under that prefix.

Why is my presigned URL sometimes redirected?

Some S3 configurations issue a 302 redirect to another endpoint. Use -L (or --location) to follow.