AWS S3 Bucket Enumeration 101: Fundamentals & Tools

Introduction

Amazon Simple Storage Service (S3) is the most widely used object store in the cloud. Because buckets are globally addressable by DNS name, they become a low-hanging fruit for reconnaissance and, when mis-configured, data leakage. This guide walks you through the core techniques for enumerating S3 buckets, locating public buckets, determining their region, and listing objects inside them.

Why does this matter? A compromised bucket can expose source code, credentials, backups, or personally identifiable information (PII). Attackers routinely start a breach by searching for “open” buckets before moving laterally. Understanding enumeration lets defenders spot the same activity in logs and harden their storage configurations.

Prerequisites

• Completed AWS Fundamentals for Security Professionals - you should know accounts, regions, and basic service concepts.
• Familiarity with IAM Permission Basics - especially the s3:ListBucket and s3:GetObject actions.
• Solid grasp of the S3 Architecture and Object Model - buckets, objects, ACLs, bucket policies, and the relationship between virtual-hosted and path-style endpoints.
• Local installation of the AWS CLI (v2) and a recent version of Python (for the open-source tools).

Core Concepts

S3 buckets are global name-spaces. When you create a bucket named my-assets, it is reachable at both:

https://my-assets.s3.amazonaws.com
https://s3.amazonaws.com/my-assets

The first form is the virtual-hosted-style endpoint; the second is the legacy path-style endpoint. Because the bucket name is part of the DNS request, an attacker can perform blind DNS queries to test for existence. Many organizations unintentionally expose buckets by leaving them world-readable or by publishing bucket names in code, logs, or CI/CD pipelines.

Key enumeration steps:

1. Discovery: Find candidate bucket names via naming conventions, DNS brute-force, or public data sources.
2. Region Resolution: Identify the AWS region that hosts the bucket - required for signed API calls.
3. Object Enumeration: List objects when permissions allow, or attempt unauthenticated reads.

Below we break each step into actionable sub-topics.

S3 bucket naming conventions and DNS-style discovery

Most organizations follow predictable naming patterns: {env}-{app}-{region}, {company}-{project}, or date-based prefixes. Understanding these conventions lets you generate candidate names quickly.

Two practical discovery methods:

• DNS enumeration: Use dig or host against the virtual-hosted endpoint. A successful NXDOMAIN means the bucket does not exist; a NOERROR response with a CNAME or A record often indicates existence.
• Search engine scraping: Indexes such as GitHub, Shodan, or public CI logs frequently contain bucket names. Simple grep on cloned repos can surface them.

Example DNS check:

host my-assets.s3.amazonaws.com
# Expected output if bucket exists
my-assets.s3.amazonaws.com is an alias for s3.amazonaws.com.

Note the use of the virtual-hosted style - the bucket name appears before .s3.amazonaws.com. When a bucket is missing, the resolver returns NXDOMAIN.

Using AWS CLI (aws s3api list-buckets) for initial enumeration

If you have valid AWS credentials, the quickest way to view buckets you own (or that your IAM principal can see) is the list-buckets API call.

aws s3api list-buckets --output json

The command returns a JSON array of bucket names and creation dates. Example output:

{ "Buckets": [ {"Name": "company-logs", "CreationDate": "2021-04-01T12:34:56.000Z"}, {"Name": "dev-static-assets", "CreationDate": "2022-10-15T08:20:30.000Z"} ], "Owner": {"DisplayName": "admin", "ID": "a1b2c3d4..."}
}

Two important points:

• The call only returns buckets that the caller’s IAM policy permits s3:ListAllMyBuckets. Most production accounts restrict this, so external attackers will not see anything.
• Even if you can list a bucket, you still need s3:ListBucket on that bucket to enumerate its objects.

For penetration testing, you can combine this with aws sts get-caller-identity to verify the credential set you are using.

Leveraging open-source tools (s3scanner, bucket-finder, goof) to locate public buckets

When you lack credentials, community tools automate the brute-force and scraping steps.

s3scanner

S3Scanner enumerates buckets by attempting HTTP HEAD requests against generated names and reports the HTTP status.

git clone GitHub repository
cd S3Scanner
python3 -m pip install -r requirements.txt
python3 s3scanner.py -w wordlist.txt -t 100

The tool prints lines such as:

[200] public-data-bucket.s3.amazonaws.com - Exists (Public List)
[403] internal-logs.s3.amazonaws.com - Exists (Access Denied)
[404] random-string123.s3.amazonaws.com - Not Found

A 200 status often means the bucket allows anonymous ListBucket, which is a high-value target.

bucket-finder

bucket-finder is a Go-based scanner that supports multi-region probing and can output CSV for further analysis.

go install GitHub repository
bucket-finder -d wordlist.txt -r us-east-1,eu-west-1 -o results.csv

Result sample:

bucket_name,region,status,public
company-static,us-east-1,200,yes
dev-configs,eu-west-1,403,no

goof

goof focuses on the “guess-the-bucket” attack using permutations of known company names and common suffixes.

go run ./cmd/goof -d "company" -s "-logs,-backup,-static" -r us-east-2

Output example:

[+] Found public bucket: company-logs-us-east-2.s3.amazonaws.com (200)
[-] No access to: company-backup-us-east-2.s3.amazonaws.com (403)

Each of these tools can be chained with jq or csvkit to produce a master list for the next phase.

Identifying bucket region and endpoint details

Knowing the region is crucial because S3 signatures are region-specific. The HEAD request to https://my-assets.s3.amazonaws.com returns an x-amz-bucket-region header when the bucket exists but the request is unsigned.

curl -I https://my-assets.s3.amazonaws.com
# Sample response headers
HTTP/1.1 200 OK
x-amz-bucket-region: us-west-2
x-amz-request-id: 1234567890ABCDEF
...

If you get 301 Moved Permanently with a Location header pointing to the correct regional endpoint, follow the redirect and read the region header.

Programmatically, the AWS CLI can resolve the region:

aws s3api get-bucket-location --bucket my-assets --output text
# Output: us-west-2

When the bucket is in the us-east-1 (the legacy “US Standard”) region, the API returns an empty string, so treat a blank response as us-east-1.

Enumerating objects within a bucket (aws s3 ls, aws s3 sync)

Once you have a bucket name and region, object enumeration depends on the bucket’s ACL or policy.

Unauthenticated enumeration

If the bucket allows s3:ListBucket to Principal: "*", a simple aws s3 ls works without credentials:

aws s3 ls s3://public-data-bucket/ --no-sign-request

Sample output:

2023-02-10 12:45:01 3420 reports/2023-02-10/report.csv
2023-02-09 09:13:12 15892 logs/app.log

Authenticated enumeration

When you possess IAM credentials that include s3:ListBucket for a target bucket, you can list recursively and even sync the entire content.

aws s3 ls s3://company-logs/ --recursive
aws s3 sync s3://company-logs/ ./company-logs-local

Both commands respect pagination and will automatically retry on throttling.

Partial enumeration via --page-size and --max-items

Large buckets may contain millions of objects. To avoid overwhelming your terminal, limit the output:

aws s3api list-objects-v2 --bucket large-bucket --max-items 1000 --page-size 500

The JSON response includes IsTruncated and NextContinuationToken fields for manual pagination.

Practical Examples

Below is a step-by-step walkthrough that combines the concepts above.

1. Generate a candidate list using a known company prefix and common suffixes.

cat > candidates.txt <<EOF
company-logs
company-backups
company-static
company-data
EOF

2. Run bucket-finder across three regions.

bucket-finder -d candidates.txt -r us-east-1,us-west-2,eu-central-1 -o bf-results.csv

3. Identify which buckets are public (status 200).

csvcut -c bucket_name,status bf-results.csv | grep ",200" | cut -d, -f1 > public-buckets.txt

4. Resolve region for each public bucket and store in a file.

while read bucket; do region=$(aws s3api get-bucket-location --bucket $bucket --output text 2>/dev/null) echo "$bucket,$region" >> public-buckets-regions.csv
done < public-buckets.txt

5. Enumerate objects without credentials (using --no-sign-request).

while IFS=, read bucket region; do echo "--- $bucket ($region) ---" aws s3 ls s3://$bucket/ --no-sign-request --recursive | head -n 10
done < <(tail -n +2 public-buckets-regions.csv)

The script prints the first ten objects of each discovered public bucket, giving you a quick data-exfiltration surface overview.

Tools & Commands

Defense & Mitigation

• Enforce least-privilege IAM policies: Remove s3:ListAllMyBuckets from users that do not need it.
• Block public access at the account and bucket level: Enable the four “Block Public Access” settings in the S3 console.
• Use bucket policies that deny s3:ListBucket to Principal: "*" unless explicitly required.
• Enable S3 Access Analyzer to surface unintended public read/write permissions.
• Implement CloudTrail data events for S3 and alert on GetObject or ListBucket from anonymous principals.
• Rename or delete unused buckets to reduce attack surface.

Common Mistakes

• Assuming a 403 response means the bucket is private. A 403 often indicates the bucket exists but you lack permission; the bucket may still be publicly listable via a different endpoint.
• Skipping region resolution. Sending signed requests to the wrong region results in 400 Bad Request - The authorization header is malformed.
• Using path-style URLs for new buckets. AWS now deprecates path-style in many regions; virtual-hosted style is required for correct region detection.
• Relying solely on list-buckets for discovery. External attackers cannot use this API without credentials.

Real-World Impact

In 2023, a well-known data breach exposed >250 GB of customer data because a backup bucket named company-backup-eu-west-1 was left world-readable. The bucket name was leaked in a public GitHub CI log, and the attacker used s3scanner to confirm the bucket’s existence, then downloaded everything with aws s3 sync --no-sign-request. The organization’s post-mortem highlighted three gaps:

1. Missing Block Public Access at the account level.
2. Absence of S3 Access Analyzer alerts for new public buckets.
3. No automated naming policy that would have forced a prefix indicating “private”.

From a defender’s view, continuous monitoring for bucket creation events combined with automated policy enforcement can stop this pattern before data leaves AWS.

Looking ahead, the rise of “data lakes” and cross-account sharing via aws s3api put-bucket-policy will increase the number of intentionally shared buckets. Security teams must therefore differentiate between legitimate sharing and accidental public exposure, leveraging tools like aws macie and s3control to enforce tagging and classification.

Practice Exercises

1. Exercise 1 - DNS enumeration: Write a Bash script that reads a list of candidate bucket names from candidates.txt and prints only those that resolve to an IP address (i.e., exist). Verify with dig and compare results to s3scanner.
2. Exercise 2 - Region detection: Using curl -I, retrieve the x-amz-bucket-region header for each public bucket discovered in Exercise 1. Store results in a CSV file.
3. Exercise 3 - Object enumeration: Pick a bucket that returned a 200 status in Exercise 1. List its first 20 objects with aws s3 ls using --no-sign-request. Capture the output and note any sensitive file extensions.
4. Exercise 4 - Defensive automation: Create a CloudWatch Event rule that triggers on PutBucketPolicy API calls. Use a Lambda function (Python) to check if the policy grants s3:* to * and send an SNS alert.

These exercises reinforce both offensive techniques and defensive automation.

Further Reading

• Amazon S3 User Guide - Bucket Permissions
• AWS Security Blog - Protecting S3 from External Attacks
• S3Scanner GitHub Repository
• AWS Macie - Sensitive Data Discovery
• OWASP Top Ten - A5: Security Misconfiguration (relevant for S3)

Summary

Enumerating Amazon S3 buckets starts with understanding the DNS-style naming model, leveraging both credentialed API calls and unauthenticated scanners, and resolving bucket regions for proper request signing. Once a bucket is identified, aws s3 ls or aws s3 sync can reveal its contents if permissions allow. Defensive best practices—blocking public access, using Access Analyzer, and monitoring bucket-policy changes—are essential to prevent data exposure. Mastery of these techniques equips security professionals to both assess risk and harden S3 deployments.