A guide: Getting started with AWS Pre-signed URLS in Software Development.

AWS Presigned URLS are temporary access grants that allow users to access private objects in Amazon S3 without requiring AWS credentials or permissions.

When to use presigned URLS?

1. Enabling File Uploads/Downloads in Client Applications

2. Serverless or Edge Computing

3. Sharing Private Content

Enabling File Uploads/Downloads in Client Applications

In some applications, downloading and uploading files to S3 through the server endpoint might be resource intensive, especially if the size of the to be uploaded or downloaded is big, it increases the payload size which might cause throttling in some APIs.

One can utilize Presigned urls to make sure file uploads and downloads doesnt affect load on the backend server. This can be done by making sure the server is only responsible for generating the presigned URL for authenticated end users, then sends the URL as a response to the frontend. Once a user clicks the link, a file upload or download will take place based on the method attached to the presigned URL.

Serverless or Edge Computing

When using AWS Lambda, the maximum payload size for a syncronous invocation endpoint is 6MB, anything bigger than that will throttle the function requests. If you are uploading a file or object to Amazon S3 using AWS Lambda, and the file size is more than 6MB, the request will throttle, returning an error like the following:
RequestEntityTooLargeException: Request payload size exceeded maximum allowed payload

For one to overcome this issue, they can utilize presigned URLS to upload and download files directly from S3, without using the lambda function’s memory or increasing any load. This is done by configuring a lambda function endpoint that will be responsible only for creating the presigned url, and returning it to the web layer(Frontend), such that when a user interacts with it, they will have a capability of downloading or uploading directly to S3 without going through the backend server(for example lambda) and without using permanent AWS access keys.

Example usage in Python (Downloading)

• A function to generate a presigned URL for downloading a file named results.pdf in a bucket named student-results and a partition final_year_2024

import boto3
from botocore.exceptions import NoCredentialsError, PartialCredentialsError
from botocore.client import Config
from decouple import config

def generate_presigned_url(bucket_name, object_key, expiration=3600):
    """
    Generates a pre-signed URL to download a file from an S3 bucket.

    :param bucket_name: Name of the S3 bucket.
    :param object_key: Key (path) of the object in the bucket.
    :param expiration: Time in seconds for the pre-signed URL to remain valid. Default is 1 hour.
    :return: Pre-signed URL as a string or an error message.
    """
    # Prompt for AWS credentials
      aws_access_key = config("AWS_ACCESS_KEY_ID")
      aws_secret_key = config("AWS_SECRET_ACCESS_KEY")

    try:
        # Initialize an S3 client with the provided credentials
        s3_client = boto3.client(
            's3',
            aws_access_key_id=aws_access_key,
            aws_secret_access_key=aws_secret_key,
            config=Config(signature_version='s3v4')
        )

        # Generate the pre-signed URL
        presigned_url = s3_client.generate_presigned_url(
            'get_object',
            Params={
                'Bucket': bucket_name,
                'Key': object_key
            },
            ExpiresIn=expiration
        )
        return presigned_url
    except NoCredentialsError:
        return "Error: No AWS credentials found."
    except PartialCredentialsError:
        return "Error: Incomplete AWS credentials found."
    except Exception as e:
        return f"An error occurred: {str(e)}"

# Parameters for the pre-signed URL
bucket_name = "student-results"
object_key = "final_year_2024/results.pdf"

The resulting url will be like:

https://student-results.s3.amazonaws.com/final_year_2024/results.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESS_KEY%2F20250109%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20250109T123456Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234

Example usage in Python (Uploading)

import boto3
from botocore.client import Config as BotoConfig
from decouple import config
from botocore.exceptions import NoCredentialsError, PartialCredentialsError

def generate_upload_url(file_name: str, bucket_name: str = "student-results", partition: str = "final_year_2024", expiration: int = 3600):
    """
    Generates a pre-signed URL for uploading a file to a specific partition in an S3 bucket.

    :param file_name: The name of the file to be uploaded.
    :param bucket_name: The name of the S3 bucket (default is "student-results").
    :param partition: The S3 partition (default is "final_year_2024").
    :param expiration: Time in seconds for the pre-signed URL to remain valid (default is 1 hour).
    :return: A pre-signed URL for uploading the file.
    """
    try:
        # Retrieve AWS credentials from environment variables using decouple
        aws_access_key = config("AWS_ACCESS_KEY_ID")
        aws_secret_key = config("AWS_SECRET_ACCESS_KEY")

        # Initialize an S3 client with the retrieved credentials
        s3_client = boto3.client(
            's3',
            aws_access_key_id=aws_access_key,
            aws_secret_access_key=aws_secret_key,
            config=BotoConfig(signature_version='s3v4')
        )

        # Set the object key (path) in the S3 bucket
        object_key = f"{partition}/{file_name}"

        # Generate the pre-signed URL for the 'put_object' operation
        presigned_url = s3_client.generate_presigned_url(
            'put_object',
            Params={
                'Bucket': bucket_name,
                'Key': object_key,
                'ContentType': 'application/pdf'  # You can modify the content type as needed
            },
            ExpiresIn=expiration
        )

        return presigned_url

    except NoCredentialsError:
        return "Error: No AWS credentials found."
    except PartialCredentialsError:
        return "Error: Incomplete AWS credentials found."
    except Exception as e:
        return f"An error occurred: {str(e)}"

# Example usage:
file_name = "new_results.pdf"
url = generate_upload_url(file_name)

The resulting url will be like:

https://student-results.s3.amazonaws.com/final_year_2024/new_results.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESS_KEY%2F20250109%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20250109T123456Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234

After the presigned urls have been returned, they can then be exposed to the frontend to be accessed by the end users , and the upload and download process wont be done by your backend server, the server is only generating the presigned URL.