AWS Day 46: Event-Driven Processing with Amazon S3 and Lambda

As part of my KodeKloud 100 Days of Cloud journey, Day 46 focused on building an event-driven file processing workflow using Amazon S3, AWS Lambda, and Amazon DynamoDB.

The objective was to automate file movement between two S3 buckets. Whenever a file was uploaded to a public S3 bucket, an S3 event notification would automatically trigger a Lambda function. The Lambda function would then copy the uploaded object to a private S3 bucket and record the operation details in DynamoDB.

This lab demonstrated how AWS managed services can be combined to create an automated workflow without requiring a continuously running server.

Step 1: Create the Public S3 Bucket

I started by opening:

AWS Console → S3 → Buckets → Create bucket

The public bucket was named:

nautilus-public-1219

The purpose of this bucket was to act as the source location for uploaded files.

The task specifically required public access to the objects in this bucket.

Step 2: Create the Private S3 Bucket

Next, I created the destination bucket:

nautilus-private-27631

This bucket was intentionally configured differently from the source bucket.

The private bucket retained:

Block all public access: Enabled

Step 3: Create the DynamoDB Table

The next component was Amazon DynamoDB.

I opened:

AWS Console → DynamoDB → Tables → Create table

The table name was:

nautilus-S3CopyLogs

The partition key was:

LogID

The data type was:

String

No sort key was required.

The resulting primary key configuration was:

Table Name : nautilus-S3CopyLogs
Partition Key : LogID
Type : String

Step 6: Configure the Lambda Function Code

The lab provided a Python file under:

/root/lambda_function.py

The function uses:

import json
import boto3
from datetime import datetime
import uuid

It initializes the AWS service clients:

s3 = boto3.client("s3")
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("nautilus-S3CopyLogs")

The DynamoDB table is therefore configured as:

nautilus-S3CopyLogs

The destination bucket is configured as:

destination_bucket = "nautilus-private-27631"

These values connect the Lambda function to the resources created earlier.

import json
import boto3
from datetime import datetime
import uuid

# Initialize the S3 and DynamoDB clients
s3 = boto3.client("s3")
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("nautilus-S3CopyLogs")


def lambda_handler(event, context):
    try:
        # Get the source bucket and object key from the event
        source_bucket = event["Records"][0]["s3"]["bucket"]["name"]
        object_key = event["Records"][0]["s3"]["object"]["key"]

        # Hardcoded destination bucket name
        destination_bucket = "nautilus-private-27631"

        # Log the event details for debugging
        print(f"[INFO] Source bucket: {source_bucket}, Object key: {object_key}")
        print(f"[INFO] Destination bucket: {destination_bucket}")

        # Copy the file from source bucket to destination bucket
        copy_source = {
            "Bucket": source_bucket,
            "Key": object_key,
        }

        print(
            f"[INFO] Attempting to copy object from {source_bucket}/{object_key} "
            f"to {destination_bucket}/{object_key}"
        )

        s3.copy_object(
            CopySource=copy_source,
            Bucket=destination_bucket,
            Key=object_key,
        )

        print(
            f"[INFO] File successfully copied from "
            f"{source_bucket}/{object_key} "
            f"to {destination_bucket}/{object_key}"
        )

        # Create log entry for DynamoDB
        log_entry = {
            "LogID": str(uuid.uuid4()),
            "SourceBucket": source_bucket,
            "DestinationBucket": destination_bucket,
            "ObjectKey": object_key,
            "Timestamp": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
            "Status": "Success",
        }

        # Log the log entry before attempting to write to DynamoDB
        print(
            f"[INFO] Writing the following log entry to DynamoDB:\n"
            f"{json.dumps(log_entry, indent=4)}"
        )

        table.put_item(Item=log_entry)

        print("[INFO] Successfully wrote log entry to DynamoDB")

        return {
            "statusCode": 200,
            "body": json.dumps(
                f"File successfully copied to {destination_bucket}"
            ),
        }

    except Exception as e:
        # Store error log in DynamoDB in case of failure
        log_entry = {
            "LogID": str(uuid.uuid4()),
            "SourceBucket": source_bucket,
            "DestinationBucket": destination_bucket,
            "ObjectKey": object_key,
            "Timestamp": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
            "Status": "Failure",
            "Error": str(e),
        }

        # Log the error log entry before attempting to write to DynamoDB
        print(
            f"[ERROR] Writing the following error log entry to DynamoDB:\n"
            f"{json.dumps(log_entry, indent=4)}"
        )

        try:
            table.put_item(Item=log_entry)
            print("[INFO] Successfully wrote error log entry to DynamoDB")
        except Exception as db_error:
            print(
                f"[ERROR] Failed to write error log entry to DynamoDB: {str(db_error)}"
            )

        # Log the error in CloudWatch
        print(f"[ERROR] Error during file copy or DynamoDB operation: {str(e)}")

        return {
            "statusCode": 500,
            "body": json.dumps(f"Error copying file: {str(e)}"),
        }

Step 7: Understand How Lambda Reads the S3 Event

When an object is uploaded to S3, the event payload contains information about the bucket and object.

The Lambda function extracts the source bucket:

source_bucket = event["Records"][0]["s3"]["bucket"]["name"]

It then extracts the object key:

object_key = event["Records"][0]["s3"]["object"]["key"]

For this lab, the uploaded object was:

sample.zip

So the Lambda function effectively receives:

Source Bucket : nautilus-public-1219
Object Key    : sample.zip

Step 9: Create the DynamoDB Log Entry

After the S3 copy succeeds, the function creates a log entry.

The code generates a unique identifier:

"LogID": str(uuid.uuid4())

It also records:

SourceBucket
DestinationBucket
ObjectKey
Timestamp
Status

The status is:

Success

The resulting DynamoDB record looks similar to:

{
  "LogID": "77d60216-0afc-45e0-bfcd-7aae031a4ae9",
  "SourceBucket": "nautilus-public-1219",
  "DestinationBucket": "nautilus-private-27631",
  "ObjectKey": "sample.zip",
  "Timestamp": "2026-07-26 14:21:04",
  "Status": "Success"
}

The function then writes it using:

table.put_item(Item=log_entry)

Final Configuration

Source S3 Bucket
Name            : nautilus-public-1219
Access          : Public as required by lab
Purpose         : File upload source

Destination S3 Bucket
Name            : nautilus-private-27631
Access          : Private
Purpose         : Secure file storage

Lambda
Function        : nautilus-copyfunction
Runtime         : Python 3.14
Execution Role  : lambda_execution_role
Trigger         : S3 Object Created

DynamoDB
Table           : nautilus-S3CopyLogs
Partition Key   : LogID
Key Type        : String

Test File
File            : sample.zip
Source          : nautilus-public-1219
Destination     : nautilus-private-27631
Status          : Success

Logging
Source Bucket      : nautilus-public-1219
Destination Bucket : nautilus-private-27631
Object Key         : sample.zip
Status              : Success

Day 46 completed: Event-driven S3 file processing with AWS Lambda, private S3 storage, IAM permissions, and DynamoDB logging.