Skip to main content

Command Palette

Search for a command to run...

4. AWS S3 Management

Published
•17 min read•View as Markdown
4. AWS S3 Management
A

🚀 Aspiring DevOps & Cloud Engineer | Passionate about Automation, CI/CD, Containers, and Cloud Infrastructure ☁️ I work with Docker, Kubernetes, Jenkins, Terraform, AWS (IAM & S3), Linux, Shell Scripting, and Git to build efficient, scalable, and secure systems. Currently contributing to DevOps-driven projects at Assurex e-Consultant while continuously expanding my skills through hands-on cloud and automation projects. Sharing my learning journey, projects, and tutorials on DevOps, AWS, and cloud technologies to help others grow in their tech careers. 💡 Let’s learn, build, and innovate together!

S3 Object Lock

Amazon S3’s Object Lock feature enables you to enforce a write-once-read-many (WORM) model, ensuring complete data immutability. By preventing objects from being permanently deleted or overwritten, Object Lock helps you meet stringent regulatory and compliance requirements.

The image describes "Object Lock," highlighting its features: preventing permanent data deletion or overwriting, meeting regulatory requirements, and enforcing a write-once-read-many (WORM) model.

How Object Lock Works

You can apply Object Lock at two levels:

  1. Object-level – Lock individual objects when you upload them.

  2. Bucket-level rule – Automatically lock every new object in the bucket.

Once locked, objects cannot be deleted or altered until the retention period expires (or until you remove a legal hold).

Use Case: Financial Records Retention

Regulated industries—such as banking and insurance—must often retain records for a defined period. With Object Lock, you can specify exactly how long data must remain immutable.

The image shows a diagram with a blue building icon on the left, an arrow labeled "5 Years" pointing to a green bucket icon on the right, under the title "Object Lock."

For example, enforcing a five-year retention period ensures that critical financial records remain tamper-proof until that timeframe ends.

Object Lock Modes

Choose one of two retention modes when locking an object:

ModeDescriptionRequired Permission
Governance ModeMost users are blocked from deleting or overwriting. Principals with bypass rights can modify.s3:BypassGovernanceRetention
Compliance ModeAll users—including the root user—are blocked from deleting or shortening retention.None (only AWS account deletion)

Note

Governance Mode lets security admins with the s3:BypassGovernanceRetention permission perform emergency deletions if needed.
Compliance Mode guarantees unbreakable WORM protection—for any removal, you must delete the entire AWS account.

The image illustrates two object lock modes: Governance Mode, where most users are restricted but users with bypass rights can access, and Compliance Mode, where all users, including root users, are restricted during the retention period.

When the exact retention period is unknown—such as during active litigation—you can apply a Legal Hold. This disables object deletion or modification indefinitely until the hold is lifted.

Only principals with the s3:PutObjectLegalHold permission can remove a Legal Hold.

aws s3api put-object-legal-hold \
  --bucket my-bucket \
  --key important-document.pdf \
  --legal-hold Status=ON

The image illustrates a "Legal Hold" process, showing that only users with "PutObjectLegalHold" permission can interact with a storage bucket, while regular users cannot. It includes a use case for documents used during active litigation.

Prerequisites

Warning

Object Lock must be enabled at bucket creation and cannot be turned on afterward.
Ensure Versioning is also enabled on the same bucket.

  • Enable Versioning on your S3 bucket.

  • Enable Object Lock when you create the bucket.

The image provides instructions for using "Object Lock," stating that versioning and object locking must be enabled in the bucket.

Once both settings are enabled, you can configure retention periods, switch modes, and apply Legal Holds to satisfy compliance mandates.

Demo S3 Object Lock

Learn how to enable and enforce Object Lock on Amazon S3 buckets, apply governance and compliance retention modes, and verify access restrictions using IAM policies.

1. Create an S3 Bucket with Object Lock

  1. In the AWS S3 console, click Create bucket.

  2. Under Advanced settings, check Enable Object Lock.

The image shows an AWS S3 bucket configuration page with options for default encryption and advanced settings, including Object Lock.

Note

Object Lock requires versioning. When you enable Object Lock, S3 automatically enables versioning for the bucket (the Versioning option is grayed out).


2. Upload an Object

Upload a test file, for example file1.txt, to your new bucket:

  1. Click Upload.

  2. Select file1.txt.

  3. Confirm and upload.

The image shows an AWS S3 Management Console screen where a file named "file1.txt" is being prepared for upload to a bucket named "kk-objectclock-demo." The file is 7.0 bytes in size and is of type "text/plain."

After upload, open the object’s Properties to configure Object Lock.


3. Configure Object Lock Retention

In the object’s Object Lock section you can choose:

  • Legal Hold: Indefinite hold without a retention date.

  • Retention Mode: Specify Governance or Compliance mode and a retention date.

Retention ModeBypass Permission RequiredUse Case
Governance Modes3:BypassGovernanceRetentionTemporary holds with exception
Compliance ModeNot bypassableRegulatory or compliance mandates

The image shows an Amazon S3 interface for editing object lock retention settings, with options for retention mode and a warning about governance mode. A specified object, "file1.txt," is listed below with details.

  1. Select Governance mode.

  2. Set the retention date (e.g., tomorrow).

  3. Click Save.

Warning

In Compliance mode, objects cannot be deleted or overwritten until the retention period expires.


4. Test Deletion with a Restricted IAM User

Switch to User Two, who has a policy denying s3:BypassGovernanceRetention. They have full S3 access but cannot bypass governance locks:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyBypassGovernanceRetention",
      "Effect": "Deny",
      "Action": "s3:BypassGovernanceRetention",
      "Resource": "*"
    }
  ]
}

When User Two tries to delete the locked object version, the request fails:

The image shows an AWS S3 console screen with a "Failed to delete objects" error message, indicating an object could not be deleted due to access denial.

User Two also cannot modify retention settings.


5. Delete with an Admin User

Switch back to User One (Administrator) with full permissions, including s3:BypassGovernanceRetention:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}

User One can now permanently delete the locked object version.


  1. Upload a second file, e.g., file2.txt.

  2. Open its Properties and scroll to Object Lock.

  3. Enable Legal Hold, then Save.

The image shows an Amazon S3 console interface displaying details of an object, including the owner, AWS region, last modified date, size, and object URL.

The object is now held indefinitely under Legal Hold.


Update User Two’s policy to also deny s3:PutObjectLegalHold, preventing removal of legal holds:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyLegalHoldAndBypass",
      "Effect": "Deny",
      "Action": [
        "s3:PutObjectLegalHold",
        "s3:BypassGovernanceRetention"
      ],
      "Resource": "*"
    }
  ]
}

Now, when User Two tries to disable the legal hold, they see a permission error:

The image shows an AWS S3 console screen where a user is attempting to edit an Object Lock legal hold but receives a permission error message.

Only users with the correct permissions (e.g., User One) can remove a legal hold.

S3 Access Logs

Amazon S3 server access logging captures detailed records for every request to a bucket, enabling you to audit access, debug issues, and optimize storage. Each operation—such as a GET on file1.txt—generates a log entry that records who accessed the object, when the request occurred, and which API call was performed:

John [06/Feb/2019:00:00:38 +0000] GET /file1.txt

Why Enable Access Logs?

Enabling server access logging provides several advantages:

BenefitDescription
Security & AuditingTrack who accessed objects and when, supporting compliance requirements.
Usage PatternsAnalyze request frequency to select the most cost-effective storage class.
Centralized Log StorageCollect logs in a dedicated bucket for easy management and analysis.

Warning

Storing and analyzing access logs may incur additional storage and request charges. Review your AWS billing dashboard to estimate costs.

What Information Is Logged?

Each log record consists of numerous fields, including:

FieldDescription
Bucket ownerAWS account ID of the bucket owner
Bucket nameName of the S3 bucket
TimestampDate and time of the request (in UTC)
Requester IP & IDClient IP address and AWS user identifier (or - if anonymous)
Request IDUnique ID assigned by S3 for the request
OperationAPI action (e.g., REST.GET.OBJECT, REST.PUT.OBJECT)
Object key & VersionRequested object path and version ID
HTTP status & ErrorHTTP response code and any error codes
Bytes sent & ReceivedPayload sizes in bytes and response timing
User agentClient application or browser details

For the complete list of log fields, see Monitoring Amazon S3 > Log Format in the AWS documentation:

The image shows a webpage from the Amazon Simple Storage Service (S3) documentation, detailing server access log formats and examples. It includes sections on HTTP status, error codes, bytes sent, object size, total time, and turn-around time.

Sample Log Entry

A typical S3 access log record begins with the bucket owner and bucket name, followed by the timestamp and requester details:

1e69f8cfa4d16e09c88f48d218e7c47fe2be DOC-EXAMPLE-BUCKET1 [06/Feb/2019:00:00:38 +0000] 192.0.2.3 79d59f4b900b49e95d5e1a69f8f8c49c8e9e09b0c8af8d218e7c47fe2be 3571 0 0

Subsequent fields describe the operation, request line, status, bytes, and user agent:

b0a88ac8fa921b2c7dfe 3E74277EXAMPLE REST.GET.VERSIONING "GET /DOC-EXAMPLE-BUCKET17/versioning HTTP/1.1" 200 113 "-" "$3Console/0.4" s9i2nYFfP76zVrKcP9<5
b0a88ac8fa921b2c7dfe 89C12E74EXAMPLE REST.GET.LOGGING_STATUS "GET /DOC-EXAMPLE-BUCKET17/logging HTTP/1.1" 200 242 11 "-" "$3Console/0.4" -9xkBeVhWfNxiM2bLxmOk0xq
b0a88ac8fa921b2c7dfe A2401f406E12 REST.GET.BUCKETPOLICY "GET /DOC-EXAMPLE-BUCKET17/policy HTTP/1.1" 404 NoSuchBucketPolicy 29 "-" "$3Console/0.4" -bMBSx0xqo
b0a88ac8fa921b2c7dfe REST.PUT.OBJECT s3-dg.pdf "PUT /DOC-EXAMPLE-BUCKET17/s3-dg.pdf HTTP/1.1" 200 - 28 "-" "$3Console/0.4" 160621Z1v8K1v8

All generated log files are delivered as plain text to your designated logging bucket:

The image shows a webpage from the AWS documentation about Amazon S3 server access log format, detailing log record fields and providing an example of log records.

Inventory

Amazon S3 Inventory provides scheduled listings of the objects in your S3 bucket, along with key metadata. You can generate these reports daily or weekly in CSV or Apache Parquet format, simplifying auditing, compliance, and analytics workflows.

What Is Amazon S3 Inventory?

An S3 Inventory report gives you a flat-file listing of all objects and their metadata stored in a bucket. It’s ideal for:

  • Auditing object-level configurations

  • Generating cost and usage analytics

  • Verifying compliance requirements

  • Bulk operations (e.g., export object keys for processing)

Report Formats and Delivery

You can choose one of two output formats:

FormatAdvantages
CSVWidely supported by spreadsheets and ETL tools
Apache ParquetOptimized for big-data queries (e.g., Amazon Athena)

Reports are delivered to a destination bucket in the same AWS Region, and you can configure optional encryption and prefix settings.

Note

Inventory reports can take up to 48 hours to appear when first enabled. Plan accordingly before running compliance checks.

Metadata Fields Included

By default, each inventory entry includes the following metadata fields:

FieldDescription
Bucket nameThe name of the source bucket
Object keyThe object’s path and file name
Version IDVersion identifier (requires versioning)
SizeObject size in bytes
Last modifiedTimestamp of the last modification
Storage classe.g., STANDARD, INTELLIGENT_TIERING
Replication statusCOMPLETE, PENDING, or FAILED
Encryption statuse.g., AES256 or AWS KMS key
Object lock statusHolds GOVERNANCE or COMPLIANCE locks

The image lists the types of information included in an inventory report, such as bucket name, key name, version ID, size, last modified date, storage class, replication status, encryption status, and object lock status. It also suggests referring to documentation for a complete list.

Refer to the AWS documentation for the full list of available fields.

Configuring S3 Inventory Reports

You can set up an inventory report via the AWS CLI, SDK, or the S3 console. Below is an example using the AWS CLI:

aws s3api put-bucket-inventory-configuration \
  --bucket my-source-bucket \
  --id daily-inventory \
  --inventory-configuration '{
    "Destination": {
      "S3BucketDestination": {
        "AccountId": "123456789012",
        "Bucket": "arn:aws:s3:::my-destination-bucket",
        "Format": "CSV",
        "Prefix": "inventory-reports/"
      }
    },
    "IsEnabled": true,
    "Filter": {
      "Prefix": "data/"
    },
    "IncludedObjectVersions": "All",
    "OptionalFields": [
      "Size",
      "LastModifiedDate",
      "StorageClass",
      "ETag",
      "IsMultipartUploaded",
      "ReplicationStatus"
    ],
    "Schedule": {
      "Frequency": "Daily"
    }
  }'

Warning

If you include VersionId, versioning must be enabled on the source bucket. Otherwise, the report will fail.

Key Configuration Options

  • Schedule.Frequency: Daily or Weekly

  • IncludedObjectVersions: All or Current

  • OptionalFields: Add any additional metadata fields you require

  • Destination.S3BucketDestination.Prefix: Organize reports under a common prefix

Access Points

Access Points streamline and secure Amazon S3 bucket access for multiple teams, applications, and workloads. Instead of a single, complex bucket policy, you can create dedicated Access Points—each with its own ARN and resource policy—to manage permissions at scale.

Note

Access Points work with existing S3 features, including bucket ACLs, public access settings, and server-side encryption.

The Challenge of Complex Bucket Policies

When you have diverse stakeholders sharing one bucket, the policy can balloon:

TeamRequired PermissionsUse Case
Developerss3:PutObject, s3:DeleteObjectUpload and manage application assets
Infrastructures3:*Full lifecycle, encryption, and logging
Legals3:GetObject, s3:ListBucketCompliance audits and data retrieval

Every change to a team’s access means editing the monolithic bucket policy, which increases the risk of errors and makes audits difficult.

Introducing Access Points

With S3 Access Points, you assign each team or application its own “window” to the bucket. Each Access Point:

  • Has a unique ARN (arn:aws:s3:<region>:<account-id>:accesspoint/<name>)

  • Behaves like an independent bucket

  • Carries a tailored resource policy

The image illustrates access points for different roles (Developers, Admin, Infra, Legal) connecting to a central bucket, likely representing a data storage or resource access system.

Clients reference the Access Point ARN instead of the bucket ARN, and permissions live closer to the consumer. For example, developers use the developer-ap Access Point ARN with write/delete privileges, while the legal team uses legal-ap with read-only permissions.

Restricting Access by VPC

You can tie an Access Point to a specific VPC Endpoint to enforce network-level boundaries. Bind the Access Point so only resources in your VPC can connect:

The image illustrates a concept of access point restriction for Virtual Private Clouds (VPCs), showing one VPC with access to a resource (indicated by a check mark) and another without access (indicated by a cross mark).

For instance:

  • VPC A: EC2 instances can access via ap-vpc-a.

  • All other VPCs: Traffic is blocked by the endpoint policy.

Warning

Ensure your VPC Endpoint policy explicitly grants s3:* actions for the Access Point ARN; otherwise, requests will be denied.

Simplifying Policy Management

Instead of embedding every role’s permissions in the bucket policy, you delegate authority to Access Points with a single bucket policy statement:

The image illustrates an "Access Point Policy" with diagrams showing the process of copying policies to a bucket policy and delegating policies to an access point.

  1. Delegate in the bucket policy to allow S3 actions for any Access Point.

  2. Define fine-grained permissions within each Access Point resource policy.

Example: Delegate List and Get operations for all Access Points in the bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DelegateAccessPointActions",
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ],
      "Condition": {
        "StringLike": {
          "aws:PrincipalArn": "arn:aws:s3:us-west-2:123456789012:accesspoint/*"
        }
      }
    }
  ]
}

Then, move user- or group-specific permissions into each Access Point policy. Example for a developer Access Point:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/Developer"
      },
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:us-west-2:123456789012:accesspoint/developer-ap/object/*"
    }
  ]
}

With this approach, you only update the bucket policy once. All subsequent permission changes happen at the Access Point level, making management and compliance audits far simpler.

Demo Access Points

In this tutorial, you’ll learn how to use Amazon S3 Access Points to delegate and isolate access control for different teams. By the end, you will have configured two access points—one for developers and one for finance each with its own fine-grained policy.

1. Create a Demo Bucket

First, set up a new S3 bucket named kk-accesspoint with the default settings. Then upload a sample file (beach.jpg) for testing.

The image shows an AWS S3 console screen where a user is configuring settings for a new bucket, including versioning, tags, and default encryption options. The "Create bucket" button is highlighted at the bottom.

Upload your test asset:

The image shows an AWS S3 upload interface where a file named "beach.jpg" is being prepared for upload. The file is 2.7 MB in size, and the "Upload" button is highlighted.

Once uploaded, as the bucket owner (user1), you can view the object details:

The image shows an Amazon S3 console page displaying details of an object named "beach.jpg," including its properties, S3 URI, and object URL. It also indicates that bucket versioning is disabled.

Best Practice

Consider enabling versioning and default encryption on production buckets to protect against accidental data loss or unauthorized access.


2. Verify Default Access for Other Users

Assume two IAM users—user2 and user3—each have only CloudShell access. By default, neither can list or retrieve objects from your new bucket.

The image shows the AWS Identity and Access Management (IAM) console, displaying a list of users with details such as last activity, password age, and active key age.

The image shows an AWS Identity and Access Management (IAM) console screen, displaying user permissions with the "AWSCloudShellFullAccess" policy attached. The console access is enabled without MFA, and no permissions boundary is set.

In AWS CloudShell, both users attempt to list and copy objects:

The image shows the AWS Management Console with a search for "CloudShell," displaying services, resources, blogs, and documentation related to AWS CloudShell. The interface is dark-themed, and there are multiple tabs open in the browser.

# As user2
[cloudshell-user@ip-... ~]$ aws s3 ls s3://kk-accesspoint/
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
[cloudshell-user@ip-... ~]$ aws s3 cp s3://kk-accesspoint/beach.jpg .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden


# As user3
[cloudshell-user@ip-... ~]$ aws s3 ls s3://kk-accesspoint/
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
[cloudshell-user@ip-... ~]$ aws s3 cp s3://kk-accesspoint/beach.jpg .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

3. Create Access Points

Navigate to Amazon S3 → Access points and create two points:

  1. developers (for user2)

  2. finance (for user3)

Select kk-accesspoint as the data source, choose Internet for Network origin, and keep public access blocking enabled.

The image shows an AWS S3 console interface for creating an access point, with fields for access point name, bucket selection, and network origin settings.

The image shows an AWS S3 Access Point configuration screen, where settings for bucket selection, AWS region, network origin, and public access blocking are being configured.

Public Access

Always keep Block all public access enabled on buckets and access points to prevent accidental exposure.


4. Delegate Bucket Permissions to Access Points

To let your access points list bucket contents, add this bucket policy. Replace 123456789012 with your AWS account ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::kk-accesspoint",
      "Condition": {
        "StringEquals": {
          "s3:DataAccessPointAccount": "123456789012"
        }
      }
    }
  ]
}

Apply under Bucket → Permissions → Bucket policy:

The image shows an Amazon S3 console screen with the "Permissions" tab open for a bucket named "kk-access-point." It displays settings related to blocking public access and bucket policies.


5. Define Access Point Policies

5.1 Developer Access Point Policy

Go to Access points → developers → Permissions → Edit and paste:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/user2"
      },
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:us-east-1:123456789012:accesspoint/developers",
        "arn:aws:s3:us-east-1:123456789012:accesspoint/developers/object/*"
      ]
    }
  ]
}

The image shows an AWS management console screen for editing an S3 Access Point policy, indicating that public access is blocked due to current settings. There are options to check and learn more about public access settings.

5.2 Finance Access Point Policy

For finance, allow user3:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/user3"
      },
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:us-east-1:123456789012:accesspoint/finance",
        "arn:aws:s3:us-east-1:123456789012:accesspoint/finance/object/*"
      ]
    }
  ]
}

After saving, review each access point’s overview:

The image shows an Amazon S3 Access Point overview page, displaying details such as bucket name, account ID, AWS region, creation date, and network origin.


Access Point Summary

Access PointPrincipalActions
developersarn:aws:iam::123456789012:user/user2List, GetObject, PutObject
financearn:aws:iam::123456789012:user/user3List, GetObject, PutObject

6. Test Access via Access Points

6.1 Developer (user2)

In AWS CloudShell as user2, list and copy via the developers access point ARN:

# List via developers access point
[cloudshell-user@... ~]$ aws s3 ls s3://arn:aws:s3:us-east-1:123456789012:accesspoint/developers
2023-09-04 07:39:25    2879314 beach.jpg


# Download the object
[cloudshell-user@... ~]$ aws s3 cp s3://arn:aws:s3:us-east-1:123456789012:accesspoint/developers/beach.jpg .

6.2 Finance (user3)

As user3, perform the same steps and upload a new file:

# List via finance access point
[cloudshell-user@... ~]$ aws s3 ls s3://arn:aws:s3:us-east-1:123456789012:accesspoint/finance
2023-09-04 07:39:25    2879314 beach.jpg


# Download the object
[cloudshell-user@... ~]$ aws s3 cp s3://arn:aws:s3:us-east-1:123456789012:accesspoint/finance/beach.jpg .


# Upload a test file
[cloudshell-user@... ~]$ touch test1
[cloudshell-user@... ~]$ aws s3 cp test1 s3://arn:aws:s3:us-east-1:123456789012:accesspoint/finance/test1


# Verify both files
[cloudshell-user@... ~]$ aws s3 ls s3://arn:aws:s3:us-east-1:123456789012:accesspoint/finance
2023-09-04 07:39:25    2879314 beach.jpg
2023-09-04 07:40:10         0 test1

7. Final Permissions Overview

Inspect the finance access point’s permissions tab:

The image shows an AWS S3 Access Point settings page, specifically the "Permissions" tab for an access point named "finance," with options to block public access enabled.


8. Conclusion

By leveraging S3 Access Points, you can:

  • Delegate access control to distinct teams without modifying the main bucket policy.

  • Create isolated entry points with tailored permissions.

  • Simplify management when multiple user groups share a bucket.

This approach improves security posture and operational efficiency in multi-team environments.