Open Source Intelligence Techniques

OSINT Techniques: Chapter 20 - API Sniffing & Web Exploitation

An exhaustive breakdown of Chapter 20 of Michael Bazzell's 'Open Source Intelligence Techniques', analyzing browser DevTools network sniffing, undocumented backend API discovery, GraphQL introspection, and automated curl/jq pipelines.

8 min read
views
visits
OSINT Techniques Chapter 20 Summary - API Sniffing & Web Exploitation
OSINT Techniques Chapter 20 Summary - API Sniffing & Web Exploitation

Welcome to our twentieth chapter analysis covering “Open Source Intelligence Techniques: Resources for Searching and Analyzing Online Information” (Sixth Edition) by Michael Bazzell.

The modern web has fundamentally shifted away from server-rendered HTML pages. Today, websites are Single Page Applications (SPAs) constructed with React, Angular, and Vue. When an investigator views a user profile, product page, or directory, the web browser downloads a lightweight skeleton page and silently executes dozens of asynchronous JavaScript background requests (AJAX / Fetch) to proprietary backend APIs.

In Chapter 20: API Sniffing & Web Exploitation, Bazzell reveals the most powerful passive intelligence technique of the modern era: intercepting and interrogating undocumented backend APIs. Front-end developers frequently rely on “security through obscurity,” writing client-side code that hides sensitive fields while the backend API happily transmits complete unmasked JSON data bundles containing private telephone numbers, GPS coordinates, internal user IDs, and administrative flags. In this comprehensive guide, we dissect DevTools Network sniffing, GraphQL introspection, and automated curl/jq intelligence pipelines.


Key Takeaways

  • The Frontend vs Backend Disconnect: The visual web page rendered on screen represents only a fraction of the data sent across the wire. Browsing with DevTools open exposes raw JSON responses before client-side scripts filter or hide them.
  • Undocumented REST Endpoints: Web applications communicate with internal microservices via undocumented REST APIs (e.g., /api/v2/internal/users/). Intercepting these requests allows analysts to replay and modify queries directly from the command line.
  • GraphQL Introspection & Schema Leaks: Many modern portals utilize GraphQL. If developers leave GraphQL Introspection enabled, sending an __schema query allows an investigator to dump the entire backend database relational schema in a single request.
  • Bypassing Client-Side Pagination & Rate Limits: While the web UI may display only 20 results per page with infinite scrolling, altering the backend API request parameter (e.g., ?limit=1000 or ?pageSize=5000) frequently dumps entire database tables without restriction.

Architectural & Investigative Workflow

The following flowchart outlines the API sniffing, reverse engineering, and automated extraction pipeline:

flowchart TD
    subgraph Browser_Interception["Phase 1: Browser DevTools Network Sniffing"]
        Browser["Target Web Application (React / Vue SPA)"] --> DevTools["Open Browser DevTools (F12 > Network Tab)"]
        DevTools --> FilterXHR["Filter Traffic by 'Fetch / XHR'"]
        FilterXHR --> InspectReq["Identify High-Value JSON Endpoints (/api/v1/profile)"]
    end
    subgraph Replay_Analysis["Phase 2: Request Deconstruction & Replay"]
        InspectReq --> CopyCurl["Right-Click Request > 'Copy as cURL'"]
        CopyCurl --> Terminal["Paste into Terminal & Analyze Auth Headers & Cookies"]
        Terminal --> ParameterTampering["Tamper with Query Parameters (limit=1000, uid=target)"]
    end
    subgraph Automated_Extraction["Phase 3: Automated Ingestion & Parsing (curl + jq)"]
        ParameterTampering --> CurlPipeline["Execute Automated Curl Request Pipeline"]
        CurlPipeline --> JQ["Pipe Raw JSON Stream into jq Filter Engine"]
        JQ --> CleanArtifacts["Extract: Unmasked Phone, Email, Roles, GPS Coordinates"]
    end
    subgraph Synthesis["Phase 4: Structured Dossier Synthesis"]
        CleanArtifacts --> TargetDossier["High-Fidelity Structured Target Profile"]
    end

Technical Breakdown & Investigative Mechanics

1. Intercepting Hidden REST APIs via Browser DevTools

To observe the undocumented conversation between the browser and backend servers:

  1. Open Firefox or Chrome and press F12 (or right-click and select Inspect).
  2. Click the Network tab and toggle the filter to Fetch/XHR.
  3. Clear existing logs, then perform an action on the website (e.g., search for a user, view a profile, or open an interactive map).
  4. Inspect the resulting HTTP requests:
    • Headers Tab: Reveals authorization tokens (Authorization: Bearer eyJ...), API keys, and session cookies.
    • Response Tab: Displays the raw, unformatted JSON payload transmitted by the server.

2. “Copy as cURL” & Building Command-Line Ingestion Pipelines

Once a high-value request is identified in DevTools:

  1. Right-click the request in the Network tab.
  2. Select Copy > Copy as cURL (bash).
  3. Paste the command into your terminal:
1
2
# Example DevTools generated cURL request
curl 'https://target-portal.com/api/v2/users/search?query=smith&limit=10'   -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) Firefox/128.0'   -H 'Accept: application/json, text/plain, */*'   -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsIn...'   --compressed

Modifying Query Parameters for Mass Exfiltration:

By tampering with the URL parameters directly:

  • Change limit=10 to limit=1000 to dump all records in a single query.
  • Remove search filters to trigger wildcard returns.
  • Change user IDs sequentially (/api/v2/users/1001, /api/v2/users/1002) to test for Broken Object Level Authorization (BOLA / IDOR).

3. Processing Complex API Responses with jq

Raw API responses often span thousands of lines of nested JSON. jq is the premier command-line JSON parser:

1
2
# Example: Extracting user names, unmasked phone numbers, and internal roles
curl -s 'https://target-portal.com/api/v2/directory'   -H 'Authorization: Bearer eyJhbGciOi...'   | jq -r '.users[] | "\(.id) | \(.full_name) | \(.email) | \(.phone_number) | \(.role)"'   | column -t -s '|'

High-Value jq Filtering Primitives:

  • jq '.[] | select(.is_admin == true)': Filters for administrative users.
  • jq '.data.locations[] | {lat: .latitude, lon: .longitude, name: .title}': Extracts spatial coordinates for instant mapping.
  • jq -r '[.users[].email] | @csv': Dumps all harvested email addresses into a clean CSV format.

4. GraphQL Introspection: Leaking the Entire Database Schema

GraphQL allows clients to request exact data structures. However, developers frequently leave Introspection Queries enabled in production, allowing an investigator to reconstruct the entire database schema:

1
2
# Querying GraphQL Introspection Endpoint via curl
curl -s -X POST 'https://target-portal.com/graphql'   -H 'Content-Type: application/json'   -d '{"query": "{ __schema { types { name fields { name } } } }"}'   | jq . > graphql_schema.json

What GraphQL Schema Leaks Reveal:

  • Hidden administrative types (e.g., AdminAuditLog, UserPasswordResetToken).
  • Unreleased features and internal staging microservice URLs.
  • Deprecated database fields that remain accessible via crafted queries.

Real-World Case Scenario

Case Study: Unmasking Hidden Seller Identifiers on a Darknet Market Mirror

An illicit vendor on an escrow marketplace operated under an anonymous pseudonym, selling stolen corporate credentials. On the web portal, the vendor’s profile displayed only their handle, feedback score, and PGP key.

An OSINT analyst inspected the web portal using DevTools:

  1. Opening the Network tab, the analyst observed a background Fetch request to:
    1
    
    GET /api/v1/vendor/profile?vendor_id=89124
    
  2. The web frontend displayed only 5 fields, but the raw JSON response payload contained 38 unredacted attributes.
  3. Inside the JSON payload, the backend had sent:
    1
    2
    3
    4
    5
    6
    7
    
    {
      "vendor_id": 89124,
      "registration_ip": "185.220.101.45",
      "recovery_email": "v_alexander_trade@proton.me",
      "telegram_uid": 981240192,
      "payout_btc_address": "bc1qxy2kgdygjrsqtzq2n0yrf2493p83kkfjhx0wlh"
    }
    
  4. The front-end React script simply did not render these fields on the screen, but the backend API delivered them freely to any authenticated client! The recovered email and BTC address unmasked the vendor immediately.

Defensive Countermeasures & OpSec Recommendations

For backend developers, API architects, and DevSecOps teams:

  1. Enforce Response Data Minimization (DTOs): Never serialize raw database models directly to JSON. Use Data Transfer Objects (DTOs) that explicitly filter out internal database IDs, IP addresses, emails, and phone numbers before transmission.
  2. Disable GraphQL Introspection in Production: Ensure introspection: false is configured in Apollo Server, GraphQL-Yoga, or Hasura before deploying to public environments.
  3. Enforce Strict Object-Level Authorization (BOLA / IDOR): Verify that the authenticated user session has explicit legal permission to view the specific requested resource ID on every API request.

Conclusion & Next Steps

Chapter 20 proves that the most critical intelligence is often transmitted directly into your browser, concealed only by the visual rendering layer. By inspecting DevTools Network streams, replaying requests via curl, and filtering JSON with jq, an investigator bypasses front-end obfuscation entirely.

In Chapter 21: Android Emulation for OSINT, we advance to mobile environments: setting up Android virtual machines, bypassing dynamic SSL certificate pinning with Frida, and intercepting mobile-only API traffic in Burp Suite.


Practical API Interception & Endpoints Discovery Cheat-Sheet

Interception Vector Command / Tool Configuration Security Outcome
GraphQL Schema Introspection curl -X POST https://target/graphql -H 'Content-Type: application/json' -d '{"query": "{__schema{types{name}}}"}' Dumps entire backend schema, exposing hidden internal endpoints and user types
Non-Proxying HTTP Sniff mitmproxy -p 8080 --anticache Captures all mobile and browser network traffic without caching stale responses
Hidden Endpoint Fuzzing ffuf -u 'https://api.target.com/v1/FUZZ' -w endpoints.txt -mc 200,401,403 Identifies hidden internal APIs, developer routes, and undocumented staging portals
Mobile API Replay curl -X GET 'https://api.target.com/user/10' -H 'Authorization: Bearer <TOKEN>' Replays intercepted mobile application API calls in terminal for automated analysis
Exposed JWT Audit jwt_tool <TOKEN> -t https://api.target.com -rh 'Authorization: Bearer' Audits JSON Web Tokens for weak HMAC secrets, none-algorithm, and privilege bypass

Authoritative References & Verified Investigative Sources


Author: Abdulrahman M. Fathi (0x3bdo) • Cyber Security Engineer & Penetration Tester

Open Research & Community Labs
صدقة جارية • Ongoing Charity

Empowering the Cybersecurity Community

All research, lab walkthroughs, and technical series on this platform remain 100% free. You can support server hosting, research environments, and lab infrastructure.

Support & Wall of Honor
اللهم اجعل هذا العمل علمًا نافعًا وصدقةً جارية، واكتب لي به الأجر والقبول، وانفع به كل من قرأه أو تعلّمه.
اللهم احفظ لي والديّ وأهلي، وبارك في أعمارهم وصحتهم، وأدم عليهم نعمة الصحة والعافية، ولا تريني فيهم بأسًا يبكيني.
Dedicated as ongoing charity and beneficial knowledge. May Allah bless my parents and family, protect them, and grant them lasting health and happiness.