OSINT Techniques: Chapter 19 - Custom Python Tools
An exhaustive breakdown of Chapter 19 of Michael Bazzell's 'Open Source Intelligence Techniques', analyzing custom Python tool authoring, asynchronous scraping architectures, rate-limit jitter, SOCKS5 proxy rotation, and structured Maltego exporting.
Welcome to our nineteenth chapter analysis covering “Open Source Intelligence Techniques: Resources for Searching and Analyzing Online Information” (Sixth Edition) by Michael Bazzell.
Commercial OSINT platforms and pre-packaged open-source tools are convenient, but they carry a fatal limitation: they are built to solve yesterday’s problems. Web applications redesign their DOM structures overnight, APIs enforce new cryptographic authentication headers, and cloud rate-limiters quickly blacklist standard user-agents. The true differentiator of an elite intelligence operative is the ability to write custom, throwaway automation scripts on the fly.
In Chapter 19: Custom Python Tools, Bazzell formalizes the development of bespoke investigative utilities. Writing a script is not just about sending an HTTP request; it requires high-throughput asynchronous I/O architectures (asyncio, aiohttp), algorithmic rate-limiting with randomized jitter, automated SOCKS5/Tor proxy rotation, resilient header spoofing, and structured data exporting. In this comprehensive guide, we dissect high-performance Python OSINT development.
Key Takeaways
- The Synchronous Bottleneck: Traditional synchronous libraries like
requestsblock execution on every network call. Scanning 10,000 potential username endpoints synchronously takes hours; deploying asynchronous event loops (asyncio) finishes the task in minutes. - Algorithmic Rate-Limit Evasion: Anti-scraping systems detect bots not merely by request frequency, but by rhythmic predictability. Injecting Poisson-distributed random delays (jitter) between requests mimics natural human browsing variance.
- SOCKS5 & Tor Proxy Multiplexing: Routing high-volume queries through rotating residential proxies or Tor circuit refreshers (
stem) prevents IP blacklisting and ensures complete operational non-attribution. - Header & Fingerprint Spoofing: Modern Cloudflare and Akamai bot protections inspect HTTP/2 settings,
Sec-Ch-Uaclient hint headers, and TLS handshake ciphers. Custom scrapers must emulate standard evergreen browser headers perfectly. - Structured Intelligence Pipelines: Outputting raw text to the terminal creates analysis paralysis; production scripts must structure harvested intelligence into JSON Lines (JSONL), SQLite databases, or Maltego-compatible entity CSVs.
Architectural & Investigative Workflow
The following flowchart outlines the architecture of an asynchronous, proxy-rotated intelligence scraping pipeline:
flowchart TD
subgraph Target_Input["Phase 1: Seed Target & Parameter Ingestion"]
TargetQueue["Target List (Usernames, Domains, Email Seeds)"] --> Dispatcher["Asyncio Worker Pool (Concurrency Limiter / Semaphore)"]
end
subgraph Evasion_Layer["Phase 2: OpSec Evasion & Proxy Routing"]
Dispatcher --> JitterEngine["Randomized Jitter & Exponential Backoff"]
JitterEngine --> HeaderSpoofer["Dynamic Header Injector (Sec-Ch-Ua, User-Agent)"]
HeaderSpoofer --> ProxyManager["SOCKS5 / Tor Proxy Circuit Manager"]
end
subgraph Request_Execution["Phase 3: Asynchronous HTTP Execution"]
ProxyManager --> AIO["aiohttp Asynchronous Session Client"]
AIO --> TargetEndpoints["External Target Endpoints (APIs / Web Portals)"]
TargetEndpoints --> ResponseHandler["Response Status & Body Evaluator"]
end
subgraph Data_Export["Phase 4: Structured Data Serialization"]
ResponseHandler --> Success{"Valid Target Match?"}
Success -- Yes --> Serializer["Structured Data Serialization Engine"]
Success -- No --> Discard["Discard / Retry Queue"]
Serializer --> JSONL["JSON Lines (JSONL) Streaming Archive"]
Serializer --> SQLite["Relational SQLite Local Intelligence DB"]
Serializer --> Maltego["Maltego Entity CSV Import"]
end
Technical Breakdown & Investigative Mechanics
1. Synchronous vs Asynchronous Architecture: The aiohttp Advantage
When conducting mass reconnaissance (e.g., verifying a target username across 500 platforms or auditing 1,000 corporate subdomains), synchronous requests is unacceptable:
1
2
3
4
5
Synchronous Requests (1 request at a time):
500 targets * 1.5 seconds average latency = 750 seconds (12.5 minutes)
Asynchronous aiohttp (50 concurrent workers):
500 targets / 50 workers = 10 batches * 1.5 seconds = 15 seconds!
2. Complete Production Asynchronous Scraper (Python CLI)
Below is an enterprise-grade asynchronous scraper implementing concurrency limits (asyncio.Semaphore), realistic browser headers, randomized jitter delays, and structured JSON output:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
#!/usr/bin/env python3
# async_osint_engine.py - High-Performance Asynchronous OSINT Engine
import asyncio
import aiohttp
import random
import json
import sys
# Realistic modern browser headers
BASE_HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.5",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1"
}
# Concurrency limit to prevent local socket exhaustion and target-side DoS
CONCURRENCY_LIMIT = 20
semaphore = asyncio.Semaphore(CONCURRENCY_LIMIT)
async def probe_endpoint(session, target_url, output_results):
async with semaphore:
# Inject randomized jitter delay (0.4 to 1.8 seconds)
jitter = random.uniform(0.4, 1.8)
await asyncio.sleep(jitter)
try:
async with session.get(target_url, headers=BASE_HEADERS, timeout=10) as response:
status = response.status
body_text = await response.text()
# Check for positive registration indicator
if status == 200 and "not found" not in body_text.lower():
result = {"url": target_url, "status": status, "live": True}
output_results.append(result)
print(f"[+] FOUND: {target_url}")
else:
print(f"[-] Absent: {target_url} (HTTP {status})")
except asyncio.TimeoutError:
print(f"[!] Timeout: {target_url}")
except Exception as e:
print(f"[!] Error querying {target_url}: {str(e)}")
async def run_pipeline(targets):
results = []
connector = aiohttp.TCPConnector(limit=50, ssl=False)
async with aiohttp.ClientSession(connector=connector) as session:
tasks = [probe_endpoint(session, url, results) for url in targets]
await asyncio.gather(*tasks)
# Serialize results to JSONL
with open("osint_harvest.jsonl", "w", encoding="utf-8") as f:
for r in results:
f.write(json.dumps(r) + "
")
print(f"
[+] Processing complete. Harvested {len(results)} confirmed endpoints.")
if __name__ == "__main__":
sample_targets = [
f"https://github.com/target_{i}" for i in range(10)
]
asyncio.run(run_pipeline(sample_targets))
3. Automated Tor Circuit Rotation via stem
When querying target platforms that rate-limit by IP address, routing Python traffic through the local Tor daemon with automated circuit renewal guarantees fresh egress nodes:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#!/usr/bin/env python3
# tor_circuit_rotator.py - Automated Tor Circuit Switching
import requests
from stem import Signal
from stem.control import Controller
import time
PROXIES = {
'http': 'socks5h://127.0.0.1:9050',
'https': 'socks5h://127.0.0.1:9050'
}
def renew_tor_circuit(control_port=9051, password='your_tor_control_password'):
with Controller.from_port(port=control_port) as controller:
controller.authenticate(password=password)
controller.signal(Signal.NEWNYM)
print("[*] Tor circuit renewed. Awaiting new IP assignment...")
time.sleep(3)
def check_current_ip():
r = requests.get('https://ipinfo.io/ip', proxies=PROXIES, timeout=10)
return r.text.strip()
# Demonstrate IP rotation
print(f"Initial Tor Exit IP: {check_current_ip()}")
renew_tor_circuit()
print(f"New Tor Exit IP: {check_current_ip()}")
4. Structuring Data for Maltego Integration
Maltego is the premier graphical link analysis tool for cyber intelligence. Custom Python scripts can output harvested intelligence directly into Maltego Entity CSV format:
1
2
3
4
5
# Maltego Entity CSV Schema Header:
# Type,Value,Properties
maltego.IPv4Address,198.51.100.25,"notes=Production Gateway"
maltego.Domain,target-corp.com,"registered=2014-05-12"
maltego.Person,"John Doe","email=jdoe@target-corp.com"
- Importing this CSV creates connected nodes in Maltego instantly, enabling graphical link analysis.
Real-World Case Scenario
Case Study: Rapid Triage of 15,000 Compromised Corporate Credentials
An adversary posted a massive 400MB text dump containing 15,000 alleged corporate email and username combinations belonging to a global aerospace firm.
The internal threat intelligence team needed to verify which credentials were real employees vs fabricated records within two hours:
- The analyst authored a custom asyncio Python script utilizing 40 worker coroutines.
- The script ingested the corporate user handles and queried the company’s public-facing Microsoft Office 365 Autodiscover and SAML authentication endpoints.
- By analyzing the timing of the server’s HTTP response headers, the script determined with 100% precision whether the account identity was active in Azure Active Directory.
- The script verified all 15,000 records in 11 minutes, discovering that only 34 accounts were active employees, allowing the SOC to instantly reset their credentials and isolate their workstations.
Defensive Countermeasures & OpSec Recommendations
For blue teams, web application firewall (WAF) administrators, and API designers:
- Implement Dynamic JA3/JA4 TLS Fingerprint Blocking: Modern bot protection tools (Cloudflare, AWS WAF, Imperva) should inspect inbound TLS Client Hello fingerprints to block default Python
urlliborrequestsTLS signatures automatically. - Deploy Sliding Window Rate-Limiting: Enforce strict token-bucket or sliding-window rate limits per IP and per session cookie on sensitive endpoints (login, password reset, profile view).
- Inject Behavioral Honeypot Fields: Place hidden input fields (
style="display:none") in HTML forms. Automated scrapers parsing DOM trees populate these fields, triggering immediate IP blacklisting.
Conclusion & Next Steps
Chapter 19 proves that custom scripting elevates an OSINT analyst from a passive user of tools to an active builder of intelligence capabilities. By mastering asynchronous execution, proxy multiplexing, and structured data serialization, an analyst can adapt to any technical obstacle.
In Chapter 20: API Sniffing & Web Exploitation, we deconstruct modern web applications: inspecting DevTools Network streams, reverse-engineering undocumented GraphQL APIs, and automating curl/jq data harvesting.
Practical Python OSINT Scripting & Automation Cheat-Sheet
| Automation Task | Python Code / Library Snippet | Operational Outcome |
|---|---|---|
| JSON API Fetching | requests.get(url, headers={'User-Agent': '...'}, timeout=10).json() |
Extracts structured JSON objects from undocumented web application endpoints |
| HTML DOM Scraping | BeautifulSoup(res.text, 'html.parser').find_all('a', href=True) |
Parses target web pages for external URLs, scripts, and embedded metadata comments |
| Bulk Email Harvesting | re.findall(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}', text) |
Mines all formatted email addresses from raw text responses with regex |
| Asynchronous Mass Probing | asyncio.gather(*[fetch(url, session) for url in url_list]) |
Executes hundreds of HTTP requests concurrently without thread overhead |
| Exporting Case Data | pandas.DataFrame(data).to_csv('evidence.csv', index=False) |
Generates structured CSV and Excel dossiers for courtroom and client presentation |
Authoritative References & Verified Investigative Sources
- Michael Bazzell, IntelTechniques: Custom Scripting and Automation for OSINT Analysts. https://inteltechniques.com/book1.html.
- Requests Library: Requests: HTTP for Humans Documentation. https://requests.readthedocs.io/.
- Beautiful Soup 4: Screen-scraping and HTML parsing with Python. https://www.crummy.com/software/BeautifulSoup/bs4/doc/.
- Scrapy Framework: Collaborative Framework for Extracting Data from Websites. https://scrapy.org/.
- SANS Institute: Automating OSINT Collection with Python. https://www.sans.org/.
Author: Abdulrahman M. Fathi (0x3bdo) • Cyber Security Engineer & Penetration Tester