Certificate Transparency Log Mining 101:...

Introduction

Certificate Transparency (CT) is a public, append-only ledger of TLS certificates issued by trusted CAs. Every certificate that complies with modern browser requirements is submitted to one or more CT logs, making the issuance process auditable and searchable.

For a security professional, CT logs are a gold-mine of reconnaissance data: they reveal newly issued certificates, sub-domains that have never been visible via DNS, and even malicious actors attempting to abuse trusted certificates for phishing or malware distribution.

Real-world examples include the Log4Shell discovery, where CT logs exposed the first wave of vulnerable Java applications, and large-scale typosquatting campaigns that were identified weeks before any DNS records were live.

Prerequisites

Solid understanding of DNS enumeration tools (dig, nslookup, host).
Familiarity with sub-domain discovery frameworks such as Amass, Subfinder, or Assetfinder.
Basic knowledge of TLS/SSL certificates - fields like SAN, CN, Issuer, NotBefore/NotAfter.
Comfort with command-line tools (curl, jq, PowerShell) and a scripting language (Python preferred).

Core Concepts

CT logs are structured as a Merkle Tree. Each leaf node contains a Signed Certificate Timestamp (SCT) that proves the certificate was logged at a specific point in time. The public API exposed by most logs (and aggregated by services like crt.sh) returns JSON objects with a fixed schema.

Key fields you will encounter:

{ "issuer_ca_id": 164, "issuer_name": "CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US", "name_value": "example.com", "min_cert_id": 12345678, "min_entry_timestamp": "2024-01-01T12:34:56Z", "not_before": "2024-01-01T12:00:00Z", "not_after": "2024-04-01T12:00:00Z", "serial_number": "04A1B2C3D4E5F6", "entry_timestamp": "2024-01-01T12:34:56Z", "source": "crtsh"
}

Understanding these fields lets you filter on issuance date, expiry, specific CAs, or wildcard domains.

CT log data model and fields

The most common model is a flat list of certificate entries. Each entry contains:

issuer_ca_id - numeric identifier of the CA (useful for filtering to private CAs).
issuer_name - full distinguished name of the issuer.
name_value - the domain(s) covered by the certificate; multiple SAN entries are concatenated with line breaks.
serial_number - hex representation, handy for cross-referencing with CT log monitors.
not_before / not_after - validity window.
entry_timestamp - when the log received the SCT.
source - usually "crtsh" when using the aggregated service.

Some logs also expose precert entries (certificates before they are fully issued). These are valuable for spotting “certificate-only” attacks where a malicious party registers a domain, obtains a cert, but never points DNS to a server.

Using crt.sh web interface and API

crt.sh provides two primary access methods:

Web UI - a simple search box where you can enter a domain, wildcard, or certificate fingerprint. Results are paginated HTML tables.
Public API - append ?output=json to any query to receive a JSON array. Example URL:
```
https://crt.sh/?q=%25example.com&output=json
```
The %25 is URL-encoded *, giving you a wildcard search.

The API is rate-limited (~10 req/s per IP) but generous for most research workloads. If you need higher throughput, rotate IPs or use the Google CT log client directly.

Automating queries with Python requests

Below is a minimal script that pulls the latest 100 certificates for a target domain and writes them to certs.json. It demonstrates pagination handling, error checking, and basic throttling.

import time, json, requests

BASE_URL = "https://crt.sh/"
QUERY = "%25example.com" # %25 = '*'
PAGE_SIZE = 100

all_certs = []
page = 0
while True: params = { "q": QUERY, "output": "json", "page": page, "limit": PAGE_SIZE } resp = requests.get(BASE_URL, params=params, timeout=15) if resp.status_code != 200: raise SystemExit(f"HTTP {resp.status_code}: {resp.text}") batch = resp.json() if not batch: break all_certs.extend(batch) print(f"Fetched {len(batch)} certs (page {page}) - total {len(all_certs)}") page += 1 time.sleep(0.2) # polite 5 req/s throttle

with open("certs.json", "w") as f: json.dump(all_certs, f, indent=2)
print("Finished - saved", len(all_certs), "records")

The script respects the public rate limit by sleeping 0.2 seconds between requests. Adjust PAGE_SIZE and time.sleep based on observed limits.

Parsing JSON results with jq and PowerShell

For quick one-liners you can avoid a full script and use jq (Linux/macOS) or PowerShell (Windows).

Extract all unique domain names using jq:

curl -s "https://crt.sh/?output=json&q=%25example.com" | jq -r '.[].name_value' | tr '
' '\0' | sort -z | uniq -z | tr '\0' '
' > domains.txt

PowerShell equivalent (using Invoke-WebRequest):

$url = "https://crt.sh/?output=json&q=%25example.com"
$response = Invoke-WebRequest -Uri $url -UseBasicParsing
$certs = $response.Content | ConvertFrom-Json
$domains = $certs | ForEach-Object { $_.name_value } | Sort-Object -Unique
$domains | Out-File -FilePath "domains.txt" -Encoding utf8

Both snippets pull the JSON, flatten the name_value field, deduplicate, and store the result for later correlation.

Correlating CT entries with DNS history

CT logs often contain domains that never resolved in DNS. To determine whether a discovered name ever existed, combine CT data with passive DNS (e.g., SecurityTrails API) or active DNS snapshots (e.g., Farsight DNSDB).

Example workflow:

Extract domain list from CT (as shown above).
Query the passive DNS API for each domain’s A/AAAA records over the last 90 days.
Mark entries with no DNS history - these are prime candidates for “certificate-only” phishing.

Python snippet using SecurityTrails (you need an API key):

import os, requests, json, time
API_KEY = os.getenv("SECURITYTRAILS_KEY")
HEADERS = {"APIKEY": API_KEY}

def has_dns_record(domain): url = f"https://api.securitytrails.com/v1/domain/{domain}/dns/history" r = requests.get(url, headers=HEADERS, timeout=10) if r.status_code != 200: return False data = r.json() return bool(data.get('records'))

with open('domains.txt') as f: domains = [d.strip() for d in f if d.strip()]

for d in domains: if not has_dns_record(d): print(f"[!] {d} - no passive DNS record") time.sleep(0.1) # avoid throttling

The script prints domains that appear only in CT, a red flag for further investigation.

Identifying suspicious certificate attributes

Not every certificate is benign. Look for red flags such as:

Wildcards on high-value TLDs (e.g., *.com, *.org) - rarely legitimate.
Short validity periods (< 30 days) - often used by fast-flux phishing kits.
Self-signed or unusual Issuer DNs - may indicate a compromised internal CA.
Mismatch between CN and SAN - attackers sometimes rely on legacy clients that ignore SAN.
Certificates issued by newly-seen CAs - check CAB Forum trusted list.

Automated scoring can be done by assigning points to each attribute and flagging anything above a threshold.

Handling pagination and rate-limit bypass

crt.sh supports page and limit parameters, but the maximum limit is 1000. When you need more than 10 k entries, you must paginate.

Techniques to stay under the limit:

Time-window slicing - break queries by entry_timestamp ranges (e.g., one-day windows).
Domain-partitioning - query a.example.com, b.example.com, etc., then merge results.
IP rotation - use a small pool of residential proxies; respect Retry-After headers.

Example of time-window pagination in Python:

import datetime, requests, json, time

BASE = "https://crt.sh/"
DOMAIN = "%25example.com"
START = datetime.datetime(2024, 1, 1)
END = datetime.datetime(2024, 1, 31)
delta = datetime.timedelta(days=1)

all_records = []
cur = START
while cur < END: nxt = cur + delta q = f"{DOMAIN}&min_entry_timestamp={cur.isoformat()}Z&max_entry_timestamp={nxt.isoformat()}Z" r = requests.get(BASE, params={"q": q, "output": "json"}) if r.status_code == 200: batch = r.json() all_records.extend(batch) print(f"Fetched {len(batch)} records for {cur.date()}") else: print("Rate limit hit, sleeping 5s") time.sleep(5) cur = nxt time.sleep(0.3)

with open('jan2024.json', 'w') as f: json.dump(all_records, f, indent=2)

By narrowing the timestamp window you avoid hitting the server-side 10 k record cap per request.

Detecting phishing/typosquatting domains

Once you have a list of CT-only domains, compare them against known brand names using fuzzy matching (Levenshtein distance) or the phishdetect library.

import Levenshtein, json

TARGET = "example.com"
THRESHOLD = 2 # edit distance

with open('domains.txt') as f: for line in f: d = line.strip() if Levenshtein.distance(d, TARGET) <= THRESHOLD: print(f"[+] Potential typo-squat: {d}")

Combine this with WHOIS lookups to see if the domain is newly registered (often < 30 days old) - a classic phishing indicator.

Integrating CT data with Shodan and Censys

Certificates are indexed by both Shodan and Censys. After you have a list of suspicious SANs, you can enrich them with host-level data (open ports, service banners, geolocation).

Shodan example (requires API key):

#!/usr/bin/env bash
API="YOUR_SHODAN_KEY"
while read -r domain; do curl -s "https://api.shodan.io/dns/domain/${domain}?key=${API}" | jq '.data[] | {ip: .ip, ports: .ports}' sleep 1
done < domains.txt

Censys provides a richer certificate search syntax. You can query directly for a fingerprint you harvested from CT:

curl -s -u "${CENSYS_UID}:${CENSYS_SECRET}" "https://search.censys.io/api/v2/certificates/search" -d '{"query": "parsed.fingerprint:AB:CD:EF:...", "page": 1, "fields": ["parsed.subject_dn", "parsed.issuer.common_name", "dns_names"]}' | jq '.'

Enriching CT findings with these platforms gives you context: is the certificate attached to a live web server? Is the host part of a known CDN or a suspicious IP range?

Practical Examples

Scenario 1 - Early Warning for a New Brand Campaign

New product acme-cloud.io is announced.
Run a nightly cron that queries crt.sh for %acme-cloud.io.
Filter for certificates issued in the last 24 h with wildcard SANs.
Cross-reference with Shodan - any host serving the cert is likely a phishing site.
Alert SOC via Slack webhook.

Python snippet for the nightly job (omits error handling for brevity):

import requests, json, datetime, os
WEBHOOK = os.getenv('SLACK_WEBHOOK')

now = datetime.datetime.utcnow().isoformat() + 'Z'
query = "%25acme-cloud.io"
url = "https://crt.sh/"
params = {"q": query, "output": "json", "min_entry_timestamp": now}
resp = requests.get(url, params=params)
certs = resp.json()
for c in certs: if "*" in c['name_value']: msg = f"*Wildcard cert detected*: {c['name_value']} issued by {c['issuer_name']}" requests.post(WEBHOOK, json={"text": msg})

This simple pipeline gives a 0-day view of misuse.

Scenario 2 - Hunting for Certificate-Only Phishing

Export all CT entries for *.login.microsoftonline.com over the past month.
Remove any entry that has a matching DNS A record (using DNSDB).
Score remaining entries for suspicious attributes (wildcards, short TTL, unknown CA).
Feed the top-scoring list into VirusTotal for URL scanning.

Result: you discover a domain login-microsoftonline.com with a valid Let's Encrypt cert but no DNS A record - a classic phishing landing page prepared for a future campaign.

Tools & Commands

curl - fetch CT JSON, Shodan, Censys APIs.
jq - lightweight JSON parser for quick filtering.
python (requests, pandas) - for bulk processing and scoring.
PowerShell - native Windows automation, especially with ConvertFrom-Json.
Amass - combine CT results with active sub-domain enumeration.
dnsx - fast DNS probing of large domain lists.
shodan CLI - enrich CT findings with host metadata.

Sample command that pulls the latest 50 certs for a target and pipes to jq for just the SAN list:

curl -s "https://crt.sh/?output=json&q=%25target.com" | jq -r '.[].name_value' | sort -u | head -n 50

Defense & Mitigation

While CT mining is a reconnaissance technique, defenders can use the same data to harden their environment:

Certificate Monitoring - set up alerts for any new cert that contains your brand name (including typos). Services like Cert Spotter or custom scripts can do this.
Domain Squatting Mitigation - pre-emptively register likely typo domains and configure them with “no-content” pages or HSTS headers.
DNS-Based Blocking - feed suspicious CT-only domains into your DNS sinkhole or firewall blocklist.
PKI Hygiene - ensure internal CAs do not issue overly permissive SANs (wildcards on public TLDs) and enforce short certificate lifetimes.

Automation is key: a nightly job that cross-references CT entries with your internal asset inventory can flag accidental exposure of internal hostnames.

Common Mistakes

Ignoring pagination - assuming a single request returns all results leads to incomplete data sets.
Not URL-encoding wildcards - using *example.com instead of %25example.com returns a 400 error.
Treating name_value as a single string - SAN entries are newline-separated; forgetting to split them loses many domains.
Over-relying on DNS resolution - many malicious actors purposefully keep the DNS record offline until the attack launch.
Missing rate-limit handling - getting blocked mid-run wastes time; always implement exponential back-off.

Real-World Impact

CT mining has been instrumental in several high-profile investigations:

Operation Phish Phry - Researchers used CT data to discover a wave of phishing sites impersonating major banks, many of which were still in the certificate-only stage.
Supply-chain compromise detection - By monitoring CT logs for a vendor’s domain, a SOC spotted a rogue certificate issued by a compromised internal CA weeks before the malicious code was deployed.
Brand protection for IoT manufacturers - CT alerts helped a device maker block counterfeit firmware update servers that had obtained valid certs for their product domain.

My experience shows that organizations that ignore CT are effectively blind to a large portion of their attack surface. As CAs move toward shorter certificate lifetimes and automated issuance, the velocity of new entries will increase, making proactive CT monitoring a must.

Practice Exercises

Basic Extraction: Write a Bash one-liner that pulls the last 200 certificates for example.org and saves only the name_value field to example_org_domains.txt.
Python Scoring Engine: Extend the provided Python script to assign a risk score (0-10) based on wildcard presence, issuer reputation, and validity window. Output a CSV with domain,score.
Enrichment Challenge: Using the Shodan CLI, enrich the top 10 high-score domains from the previous exercise with ip,port,org data and produce a markdown table.
Time-Window Pagination: Implement a Go or Rust program that queries CT logs for a full year, splitting the request by month, and writes each month’s JSON to a separate file.

These labs reinforce API usage, pagination, data enrichment, and risk modeling.

Summary

CT logs expose every publicly trusted TLS certificate - a rich reconnaissance source.
crt.sh provides a simple JSON API; pagination and rate-limit handling are essential for large queries.
Automate extraction with Python requests, parse with jq or PowerShell, and enrich with passive DNS, Shodan, or Censys.
Look for suspicious attributes (wildcards, short lifetimes, unknown CAs) and correlate with DNS history to flag certificate-only phishing.
Integrate CT monitoring into your brand-protection and incident-response workflows for early detection.

Certificate Transparency Log Mining 101: Fundamentals and Data Sources