OSINT Techniques: Chapter 20 - API Sniffing & Web Exploitation
An exhaustive breakdown of Chapter 20 of Michael Bazzell's 'Open Source Intelligence Techniques', analyzing browser DevTools network sniffing, undocumented backend API discovery, GraphQL introspection, and automated curl/jq pipelines.
Welcome to our twentieth chapter analysis covering “Open Source Intelligence Techniques: Resources for Searching and Analyzing Online Information” (Sixth Edition) by Michael Bazzell.
The modern web has fundamentally shifted away from server-rendered HTML pages. Today, websites are Single Page Applications (SPAs) constructed with React, Angular, and Vue. When an investigator views a user profile, product page, or directory, the web browser downloads a lightweight skeleton page and silently executes dozens of asynchronous JavaScript background requests (AJAX / Fetch) to proprietary backend APIs.
In Chapter 20: API Sniffing & Web Exploitation, Bazzell reveals the most powerful passive intelligence technique of the modern era: intercepting and interrogating undocumented backend APIs. Front-end developers frequently rely on “security through obscurity,” writing client-side code that hides sensitive fields while the backend API happily transmits complete unmasked JSON data bundles containing private telephone numbers, GPS coordinates, internal user IDs, and administrative flags. In this comprehensive guide, we dissect DevTools Network sniffing, GraphQL introspection, and automated curl/jq intelligence pipelines.
Key Takeaways
- The Frontend vs Backend Disconnect: The visual web page rendered on screen represents only a fraction of the data sent across the wire. Browsing with DevTools open exposes raw JSON responses before client-side scripts filter or hide them.
- Undocumented REST Endpoints: Web applications communicate with internal microservices via undocumented REST APIs (e.g.,
/api/v2/internal/users/). Intercepting these requests allows analysts to replay and modify queries directly from the command line. - GraphQL Introspection & Schema Leaks: Many modern portals utilize GraphQL. If developers leave GraphQL Introspection enabled, sending an
__schemaquery allows an investigator to dump the entire backend database relational schema in a single request. - Bypassing Client-Side Pagination & Rate Limits: While the web UI may display only 20 results per page with infinite scrolling, altering the backend API request parameter (e.g.,
?limit=1000or?pageSize=5000) frequently dumps entire database tables without restriction.
Architectural & Investigative Workflow
The following flowchart outlines the API sniffing, reverse engineering, and automated extraction pipeline:
flowchart TD
subgraph Browser_Interception["Phase 1: Browser DevTools Network Sniffing"]
Browser["Target Web Application (React / Vue SPA)"] --> DevTools["Open Browser DevTools (F12 > Network Tab)"]
DevTools --> FilterXHR["Filter Traffic by 'Fetch / XHR'"]
FilterXHR --> InspectReq["Identify High-Value JSON Endpoints (/api/v1/profile)"]
end
subgraph Replay_Analysis["Phase 2: Request Deconstruction & Replay"]
InspectReq --> CopyCurl["Right-Click Request > 'Copy as cURL'"]
CopyCurl --> Terminal["Paste into Terminal & Analyze Auth Headers & Cookies"]
Terminal --> ParameterTampering["Tamper with Query Parameters (limit=1000, uid=target)"]
end
subgraph Automated_Extraction["Phase 3: Automated Ingestion & Parsing (curl + jq)"]
ParameterTampering --> CurlPipeline["Execute Automated Curl Request Pipeline"]
CurlPipeline --> JQ["Pipe Raw JSON Stream into jq Filter Engine"]
JQ --> CleanArtifacts["Extract: Unmasked Phone, Email, Roles, GPS Coordinates"]
end
subgraph Synthesis["Phase 4: Structured Dossier Synthesis"]
CleanArtifacts --> TargetDossier["High-Fidelity Structured Target Profile"]
end
Technical Breakdown & Investigative Mechanics
1. Intercepting Hidden REST APIs via Browser DevTools
To observe the undocumented conversation between the browser and backend servers:
- Open Firefox or Chrome and press
F12(or right-click and select Inspect). - Click the Network tab and toggle the filter to Fetch/XHR.
- Clear existing logs, then perform an action on the website (e.g., search for a user, view a profile, or open an interactive map).
- Inspect the resulting HTTP requests:
- Headers Tab: Reveals authorization tokens (
Authorization: Bearer eyJ...), API keys, and session cookies. - Response Tab: Displays the raw, unformatted JSON payload transmitted by the server.
- Headers Tab: Reveals authorization tokens (
2. “Copy as cURL” & Building Command-Line Ingestion Pipelines
Once a high-value request is identified in DevTools:
- Right-click the request in the Network tab.
- Select Copy > Copy as cURL (bash).
- Paste the command into your terminal:
1
2
# Example DevTools generated cURL request
curl 'https://target-portal.com/api/v2/users/search?query=smith&limit=10' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) Firefox/128.0' -H 'Accept: application/json, text/plain, */*' -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsIn...' --compressed
Modifying Query Parameters for Mass Exfiltration:
By tampering with the URL parameters directly:
- Change
limit=10tolimit=1000to dump all records in a single query. - Remove search filters to trigger wildcard returns.
- Change user IDs sequentially (
/api/v2/users/1001,/api/v2/users/1002) to test for Broken Object Level Authorization (BOLA / IDOR).
3. Processing Complex API Responses with jq
Raw API responses often span thousands of lines of nested JSON. jq is the premier command-line JSON parser:
1
2
# Example: Extracting user names, unmasked phone numbers, and internal roles
curl -s 'https://target-portal.com/api/v2/directory' -H 'Authorization: Bearer eyJhbGciOi...' | jq -r '.users[] | "\(.id) | \(.full_name) | \(.email) | \(.phone_number) | \(.role)"' | column -t -s '|'
High-Value jq Filtering Primitives:
jq '.[] | select(.is_admin == true)': Filters for administrative users.jq '.data.locations[] | {lat: .latitude, lon: .longitude, name: .title}': Extracts spatial coordinates for instant mapping.jq -r '[.users[].email] | @csv': Dumps all harvested email addresses into a clean CSV format.
4. GraphQL Introspection: Leaking the Entire Database Schema
GraphQL allows clients to request exact data structures. However, developers frequently leave Introspection Queries enabled in production, allowing an investigator to reconstruct the entire database schema:
1
2
# Querying GraphQL Introspection Endpoint via curl
curl -s -X POST 'https://target-portal.com/graphql' -H 'Content-Type: application/json' -d '{"query": "{ __schema { types { name fields { name } } } }"}' | jq . > graphql_schema.json
What GraphQL Schema Leaks Reveal:
- Hidden administrative types (e.g.,
AdminAuditLog,UserPasswordResetToken). - Unreleased features and internal staging microservice URLs.
- Deprecated database fields that remain accessible via crafted queries.
Real-World Case Scenario
Case Study: Unmasking Hidden Seller Identifiers on a Darknet Market Mirror
An illicit vendor on an escrow marketplace operated under an anonymous pseudonym, selling stolen corporate credentials. On the web portal, the vendor’s profile displayed only their handle, feedback score, and PGP key.
An OSINT analyst inspected the web portal using DevTools:
- Opening the Network tab, the analyst observed a background Fetch request to:
1 GET /api/v1/vendor/profile?vendor_id=89124- The web frontend displayed only 5 fields, but the raw JSON response payload contained 38 unredacted attributes.
- Inside the JSON payload, the backend had sent:
1 2 3 4 5 6 7 { "vendor_id": 89124, "registration_ip": "185.220.101.45", "recovery_email": "v_alexander_trade@proton.me", "telegram_uid": 981240192, "payout_btc_address": "bc1qxy2kgdygjrsqtzq2n0yrf2493p83kkfjhx0wlh" }- The front-end React script simply did not render these fields on the screen, but the backend API delivered them freely to any authenticated client! The recovered email and BTC address unmasked the vendor immediately.
Defensive Countermeasures & OpSec Recommendations
For backend developers, API architects, and DevSecOps teams:
- Enforce Response Data Minimization (DTOs): Never serialize raw database models directly to JSON. Use Data Transfer Objects (DTOs) that explicitly filter out internal database IDs, IP addresses, emails, and phone numbers before transmission.
- Disable GraphQL Introspection in Production: Ensure
introspection: falseis configured in Apollo Server, GraphQL-Yoga, or Hasura before deploying to public environments. - Enforce Strict Object-Level Authorization (BOLA / IDOR): Verify that the authenticated user session has explicit legal permission to view the specific requested resource ID on every API request.
Conclusion & Next Steps
Chapter 20 proves that the most critical intelligence is often transmitted directly into your browser, concealed only by the visual rendering layer. By inspecting DevTools Network streams, replaying requests via curl, and filtering JSON with jq, an investigator bypasses front-end obfuscation entirely.
In Chapter 21: Android Emulation for OSINT, we advance to mobile environments: setting up Android virtual machines, bypassing dynamic SSL certificate pinning with Frida, and intercepting mobile-only API traffic in Burp Suite.
Practical API Interception & Endpoints Discovery Cheat-Sheet
| Interception Vector | Command / Tool Configuration | Security Outcome |
|---|---|---|
| GraphQL Schema Introspection | curl -X POST https://target/graphql -H 'Content-Type: application/json' -d '{"query": "{__schema{types{name}}}"}' |
Dumps entire backend schema, exposing hidden internal endpoints and user types |
| Non-Proxying HTTP Sniff | mitmproxy -p 8080 --anticache |
Captures all mobile and browser network traffic without caching stale responses |
| Hidden Endpoint Fuzzing | ffuf -u 'https://api.target.com/v1/FUZZ' -w endpoints.txt -mc 200,401,403 |
Identifies hidden internal APIs, developer routes, and undocumented staging portals |
| Mobile API Replay | curl -X GET 'https://api.target.com/user/10' -H 'Authorization: Bearer <TOKEN>' |
Replays intercepted mobile application API calls in terminal for automated analysis |
| Exposed JWT Audit | jwt_tool <TOKEN> -t https://api.target.com -rh 'Authorization: Bearer' |
Audits JSON Web Tokens for weak HMAC secrets, none-algorithm, and privilege bypass |
Authoritative References & Verified Investigative Sources
- Michael Bazzell, IntelTechniques: API Sniffing and Web Exploration for Investigative Intelligence. https://inteltechniques.com/book1.html.
- OWASP Foundation: OWASP API Security Top 10 Project. https://owasp.org/www-project-api-security/.
- mitmproxy Project: An interactive, SSL/TLS-capable intercepting HTTP proxy. https://mitmproxy.org/.
- PortSwigger Web Security: API Testing & Information Disclosure Methodologies. https://portswigger.net/web-security/api-testing.
- GraphQL Foundation: GraphQL Security Testing Best Practices & Introspection Guidelines. https://graphql.org/learn/best-practices/.
Author: Abdulrahman M. Fathi (0x3bdo) • Cyber Security Engineer & Penetration Tester