Open Source Intelligence Techniques

OSINT Techniques: Chapter 13 - Documents & Metadata

An exhaustive breakdown of Chapter 13 of Michael Bazzell's 'Open Source Intelligence Techniques', analyzing document metadata, ExifTool forensics, PDF object stream carving, and recovering flawed black-box redactions.

9 min read
views
visits
OSINT Techniques Chapter 13 Summary - Documents & Metadata
OSINT Techniques Chapter 13 Summary - Documents & Metadata

Welcome to our thirteenth chapter analysis covering “Open Source Intelligence Techniques: Resources for Searching and Analyzing Online Information” (Sixth Edition) by Michael Bazzell.

When organizations publish whitepapers, press releases, financial statements, or internal memos, they believe they are sharing only the visible text and graphics rendered on the page. In reality, modern document formats—such as Adobe PDF and Microsoft Office Open XML (.docx, .xlsx)—are complex container archives encapsulating embedded metadata, internal network file paths, author usernames, printer serial numbers, software compilation versions, and revision histories.

In Chapter 13: Documents & Metadata, Bazzell details the forensic examination of digital files. An investigator does not merely read a document; they dissect its binary headers and XML schema. In this comprehensive guide, we master Phil Harvey’s ExifTool, analyze automated document collectors (FOCA), perform low-level PDF object stream carving with Didier Stevens’ tools, and expose the catastrophic security failures of amateur PDF redactions.


Key Takeaways

  • The Invisibility of Metadata: Document creation software silently injects author names, operating system login handles, internal file share paths (\corp-fs01\legal\cases\), and hardware GUIDs directly into file structures.
  • ExifTool as the Forensic Standard: Phil Harvey’s ExifTool remains the undisputed gold standard for metadata extraction and sanitization, supporting hundreds of file formats across EXIF, XMP, IPTC, and PDF dictionaries.
  • The Black-Box Redaction Fallacy: Drawing a black rectangle over sensitive text in standard PDF editors (like Adobe Acrobat or Preview) merely places a vector graphic overlay on top of the text; the underlying ASCII/Unicode characters remain completely intact and extractable via simple copy-paste or text scraping.
  • PDF Object Stream Forensics: PDF files are organized as numbered object streams (obj ... endobj). Utilizing tools like pdfid and pdf-parser allows analysts to discover hidden JavaScript, incremental revision histories, and deleted text retained in slack space.

Architectural & Investigative Workflow

The following flowchart outlines the document metadata harvesting and forensic inspection pipeline:

flowchart TD
    subgraph Document_Acquisition["Phase 1: Automated Document Harvesting"]
        TargetDomain["Target Enterprise Domain (target.com)"] --> Dorks["Google Dorking for Public Documents (filetype:pdf, docx, xlsx)"]
        Dorks --> FOCA["FOCA / Metagoofil Automated Document Harvester"]
        FOCA --> Corpus["Corpus of Downloaded Target Documents"]
    end
    subgraph Metadata_Extraction["Phase 2: Deep Metadata Extraction"]
        Corpus --> ExifTool["ExifTool Forensic Extraction (-a -u -g1 -s)"]
        ExifTool --> Attributes["Extract: Authors, Software, Timestamps, File Paths"]
        Attributes --> OrgStructure["Map Internal Network Paths & Username Conventions"]
    end
    subgraph Stream_Analysis["Phase 3: PDF Object & Redaction Forensics"]
        Corpus --> PDFID["Didier Stevens pdfid & pdf-parser Inspection"]
        PDFID --> Redactions["Audit Black-Box Overlays for Underlying Text"]
        PDFID --> Revisions["Extract Incremental Updates & Deleted Drafts"]
    end
    subgraph Synthesis["Phase 4: Threat Intelligence Synthesis"]
        OrgStructure --> TargetProfile["Internal Attack Surface & Key Personnel Dossier"]
        Redactions --> TargetProfile
        Revisions --> TargetProfile
    end

Technical Breakdown & Investigative Mechanics

1. Mastering ExifTool for In-Depth Metadata Extraction

While basic GUI properties dialogs hide 90% of metadata tags, the command-line utility ExifTool interrogates every byte:

1
2
3
# Extract ALL metadata tags, including unknown (-u), duplicate (-a), 
# organized by family group (-g1), and displaying tag IDs (-s)
exiftool -a -u -g1 -s target_document.pdf

Key Forensic Tags to Scrutinize:

  • [PDF] Creator: The specific application that authored the original document (e.g., Microsoft Word for Office 365).
  • [PDF] Producer: The rendering engine that converted the document into PDF (e.g., macOS Version 14.5 Quartz PDFContext).
  • [XMP-xmp] CreatorTool: Internal software build strings.
  • [XMP-xmpMM] DocumentID: Unique UUID tracking the document lifecycle across edits.
  • [System] FileModifyDate: Server timestamp when the file was last saved.

Batch Harvesting Metadata from an Entire Folder:

1
2
# Scan an entire directory of documents and export structured CSV intelligence
exiftool -csv -r -Author -Creator -Producer -CreateDate -ModifyDate ./evidence_docs/ > metadata_report.csv

2. Dissecting Microsoft Office Open XML (.docx / .xlsx)

Modern Microsoft Office files are standard ZIP archives containing XML trees. When ExifTool indicates an interesting Word file, an analyst can unpack it directly:

1
2
3
4
5
6
7
8
# Rename and decompress the DOCX archive
unzip target_memo.docx -d ./unpacked_doc/

# Inspect core metadata properties
cat ./unpacked_doc/docProps/core.xml | jq .

# Inspect internal application settings and total editing duration
cat ./unpacked_doc/docProps/app.xml

What app.xml and core.xml Reveal:

  • TotalEditingTime: Minutes spent editing the document (reveals if a statement was rushed or carefully drafted).
  • Company: Registered enterprise organization name in Microsoft Office license.
  • LastModifiedBy: Username of the last person who saved the file (frequently exposes administrative accounts).

3. PDF Object Stream Forensics with pdfid and pdf-parser

PDF files support incremental updates: when a user edits or deletes text and saves, the PDF engine appends a new revision to the end of the file rather than rewriting the binary.

1
2
3
4
5
6
# Installing Didier Stevens PDF forensic tools
sudo apt-get install -y python3-pip
pip install pdf-parser pdfid

# Step 1: Scan PDF for suspicious object types and incremental streams
pdfid target_confidential.pdf

Sample pdfid Output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
PDFiD 0.2.8 target_confidential.pdf
 PDF Header: %PDF-1.7
 obj                   142
 endobj                142
 stream                 28
 endstream              28
 xref                    2
 trailer                 2
 startxref               2
 /Page                   4
 /Encrypt                0
 /ObjStm                 0
 /JS                     0
 /JavaScript             0
 /AA                     0
 /OpenAction             0
 /AcroForm               0

Extracting Text from Incremental Revisions via pdf-parser:

1
2
3
4
5
# Search for all text objects inside the PDF stream
pdf-parser --search "CONFIDENTIAL" target_confidential.pdf

# Dump raw content of object stream #12 to inspect previous revisions
pdf-parser --object 12 --raw --filter target_confidential.pdf

4. The Flawed Redaction Exploit

One of the most persistent operational blunders in government and corporate disclosures is improper redaction:

graph TD
    A["User Opens PDF in Adobe Reader / Graphic Software"] --> B["Draws Black Filled Rectangle over Secret Names"]
    B --> C["Clicks 'Save' (Believes text is permanently destroyed)"]
    C --> D["Analyst Opens PDF in Poppler (pdftotext) or Selects All"]
    D --> E["Black Box is merely a Vector Shape Layer (Z-index top)"]
    E --> F["Full Plaintext Extracted Instantly from Background Layer!"]

Testing for Flawed Redactions via Command Line:

1
2
3
4
5
# Extract all raw underlying plaintext using poppler-utils
pdftotext -layout target_redacted.pdf extracted_plaintext.txt

# Inspect extracted text for previously 'blacked-out' passages
grep -i "settlement amount" extracted_plaintext.txt

Real-World Case Scenario

Case Study: Uncovering an Anonymous Whistleblower’s True Identity

An anonymous whistleblower sent a 12-page PDF dossier alleging environmental corruption by a chemical manufacturer to several major newspapers.

While the text was written from an external perspective, the recipient media group asked an OSINT analyst to verify the document’s authenticity:

  1. Running ExifTool revealed the Author tag was blank, but the Producer tag read: macOS Version 14.2 Quartz PDFContext.
  2. Decompressing the PDF streams via pdf-parser located an embedded thumbnail preview image (Obj 18).
  3. Extracting the thumbnail revealed a full-screen screenshot of an Apple Mail inbox showing the logged-in email address of the company’s Vice President of Regulatory Affairs.
  4. Furthermore, on page 8, a black rectangle had been placed over an internal project code. Running pdftotext extracted the underlying string: Project Blue-Horizon Phase 3.

Defensive Countermeasures & OpSec Recommendations

For corporate legal teams, privacy officers, and whistleblowers:

  1. Deploy True Redaction Tools: Never use simple highlighting or rectangle drawing tools to redact. Always use dedicated redaction features (e.g., Adobe Acrobat Pro Sanitize Document or Apply Redactions) that permanently rasterize and destroy underlying vector glyphs.
  2. Automate CI/CD Metadata Sanitization: Enforce an automated gateway script that executes exiftool -all= -overwrite_original on all public documents prior to publishing on public websites.
  3. Convert to Clean Bitmap (Rasterization): If absolute privacy is required, convert the PDF into high-resolution PNG images (pdftoppm -png), visually inspect the pixels, and recompile into a clean PDF (img2pdf).

Conclusion & Next Steps

Chapter 13 proves that documents are rich forensic artifacts that frequently reveal more about their creators than the text intended for publication. By mastering ExifTool, parsing PDF object streams, and testing for redaction flaws, an investigator can uncover internal organizational truths.

In Chapter 14: Photographs & Reverse Image Search, we advance to visual forensic intelligence: camera sensor PRNU fingerprinting, Error Level Analysis (ELA), and cutting-edge facial recognition engines.


Practical Document Forensics & Metadata Extraction Cheat-Sheet

Forensic Operation Terminal Command / Tool Investigative Benefit
Comprehensive Metadata Dump exiftool -a -u -g1 target.pdf Extracts author names, operating systems, printer serials, and revision timestamps
Extract Embedded Attachments pdfdetach -list target.pdf && pdfdetach -saveall target.pdf Carves hidden binary attachments, spreadsheets, and scripts embedded in PDF objects
Automated Document Harvester foca --domain target.com --extensions pdf,docx,xlsx Crawls target domain, extracts all public documents, and builds organizational IT profile
Decompress Office XML unzip -p target.docx docProps/core.xml | xmllint --format - Extracts raw creator, lastModifiedBy, and revision count directly from XML headers
Strip Metadata for OpSec exiftool -all= target.jpg Zeroes out all internal EXIF, XMP, and IPTC headers prior to public file release

Authoritative References & Verified Investigative Sources


Author: Abdulrahman M. Fathi (0x3bdo) • Cyber Security Engineer & Penetration Tester

Open Research & Community Labs
صدقة جارية • Ongoing Charity

Empowering the Cybersecurity Community

All research, lab walkthroughs, and technical series on this platform remain 100% free. You can support server hosting, research environments, and lab infrastructure.

Support & Wall of Honor
اللهم اجعل هذا العمل علمًا نافعًا وصدقةً جارية، واكتب لي به الأجر والقبول، وانفع به كل من قرأه أو تعلّمه.
اللهم احفظ لي والديّ وأهلي، وبارك في أعمارهم وصحتهم، وأدم عليهم نعمة الصحة والعافية، ولا تريني فيهم بأسًا يبكيني.
Dedicated as ongoing charity and beneficial knowledge. May Allah bless my parents and family, protect them, and grant them lasting health and happiness.