OSINT Techniques: Chapter 13 - Documents & Metadata
An exhaustive breakdown of Chapter 13 of Michael Bazzell's 'Open Source Intelligence Techniques', analyzing document metadata, ExifTool forensics, PDF object stream carving, and recovering flawed black-box redactions.
Welcome to our thirteenth chapter analysis covering “Open Source Intelligence Techniques: Resources for Searching and Analyzing Online Information” (Sixth Edition) by Michael Bazzell.
When organizations publish whitepapers, press releases, financial statements, or internal memos, they believe they are sharing only the visible text and graphics rendered on the page. In reality, modern document formats—such as Adobe PDF and Microsoft Office Open XML (.docx, .xlsx)—are complex container archives encapsulating embedded metadata, internal network file paths, author usernames, printer serial numbers, software compilation versions, and revision histories.
In Chapter 13: Documents & Metadata, Bazzell details the forensic examination of digital files. An investigator does not merely read a document; they dissect its binary headers and XML schema. In this comprehensive guide, we master Phil Harvey’s ExifTool, analyze automated document collectors (FOCA), perform low-level PDF object stream carving with Didier Stevens’ tools, and expose the catastrophic security failures of amateur PDF redactions.
Key Takeaways
- The Invisibility of Metadata: Document creation software silently injects author names, operating system login handles, internal file share paths (
\corp-fs01\legal\cases\), and hardware GUIDs directly into file structures. - ExifTool as the Forensic Standard: Phil Harvey’s ExifTool remains the undisputed gold standard for metadata extraction and sanitization, supporting hundreds of file formats across EXIF, XMP, IPTC, and PDF dictionaries.
- The Black-Box Redaction Fallacy: Drawing a black rectangle over sensitive text in standard PDF editors (like Adobe Acrobat or Preview) merely places a vector graphic overlay on top of the text; the underlying ASCII/Unicode characters remain completely intact and extractable via simple copy-paste or text scraping.
- PDF Object Stream Forensics: PDF files are organized as numbered object streams (
obj ... endobj). Utilizing tools likepdfidandpdf-parserallows analysts to discover hidden JavaScript, incremental revision histories, and deleted text retained in slack space.
Architectural & Investigative Workflow
The following flowchart outlines the document metadata harvesting and forensic inspection pipeline:
flowchart TD
subgraph Document_Acquisition["Phase 1: Automated Document Harvesting"]
TargetDomain["Target Enterprise Domain (target.com)"] --> Dorks["Google Dorking for Public Documents (filetype:pdf, docx, xlsx)"]
Dorks --> FOCA["FOCA / Metagoofil Automated Document Harvester"]
FOCA --> Corpus["Corpus of Downloaded Target Documents"]
end
subgraph Metadata_Extraction["Phase 2: Deep Metadata Extraction"]
Corpus --> ExifTool["ExifTool Forensic Extraction (-a -u -g1 -s)"]
ExifTool --> Attributes["Extract: Authors, Software, Timestamps, File Paths"]
Attributes --> OrgStructure["Map Internal Network Paths & Username Conventions"]
end
subgraph Stream_Analysis["Phase 3: PDF Object & Redaction Forensics"]
Corpus --> PDFID["Didier Stevens pdfid & pdf-parser Inspection"]
PDFID --> Redactions["Audit Black-Box Overlays for Underlying Text"]
PDFID --> Revisions["Extract Incremental Updates & Deleted Drafts"]
end
subgraph Synthesis["Phase 4: Threat Intelligence Synthesis"]
OrgStructure --> TargetProfile["Internal Attack Surface & Key Personnel Dossier"]
Redactions --> TargetProfile
Revisions --> TargetProfile
end
Technical Breakdown & Investigative Mechanics
1. Mastering ExifTool for In-Depth Metadata Extraction
While basic GUI properties dialogs hide 90% of metadata tags, the command-line utility ExifTool interrogates every byte:
1
2
3
# Extract ALL metadata tags, including unknown (-u), duplicate (-a),
# organized by family group (-g1), and displaying tag IDs (-s)
exiftool -a -u -g1 -s target_document.pdf
Key Forensic Tags to Scrutinize:
[PDF] Creator: The specific application that authored the original document (e.g., Microsoft Word for Office 365).[PDF] Producer: The rendering engine that converted the document into PDF (e.g., macOS Version 14.5 Quartz PDFContext).[XMP-xmp] CreatorTool: Internal software build strings.[XMP-xmpMM] DocumentID: Unique UUID tracking the document lifecycle across edits.[System] FileModifyDate: Server timestamp when the file was last saved.
Batch Harvesting Metadata from an Entire Folder:
1
2
# Scan an entire directory of documents and export structured CSV intelligence
exiftool -csv -r -Author -Creator -Producer -CreateDate -ModifyDate ./evidence_docs/ > metadata_report.csv
2. Dissecting Microsoft Office Open XML (.docx / .xlsx)
Modern Microsoft Office files are standard ZIP archives containing XML trees. When ExifTool indicates an interesting Word file, an analyst can unpack it directly:
1
2
3
4
5
6
7
8
# Rename and decompress the DOCX archive
unzip target_memo.docx -d ./unpacked_doc/
# Inspect core metadata properties
cat ./unpacked_doc/docProps/core.xml | jq .
# Inspect internal application settings and total editing duration
cat ./unpacked_doc/docProps/app.xml
What app.xml and core.xml Reveal:
TotalEditingTime: Minutes spent editing the document (reveals if a statement was rushed or carefully drafted).Company: Registered enterprise organization name in Microsoft Office license.LastModifiedBy: Username of the last person who saved the file (frequently exposes administrative accounts).
3. PDF Object Stream Forensics with pdfid and pdf-parser
PDF files support incremental updates: when a user edits or deletes text and saves, the PDF engine appends a new revision to the end of the file rather than rewriting the binary.
1
2
3
4
5
6
# Installing Didier Stevens PDF forensic tools
sudo apt-get install -y python3-pip
pip install pdf-parser pdfid
# Step 1: Scan PDF for suspicious object types and incremental streams
pdfid target_confidential.pdf
Sample pdfid Output:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
PDFiD 0.2.8 target_confidential.pdf
PDF Header: %PDF-1.7
obj 142
endobj 142
stream 28
endstream 28
xref 2
trailer 2
startxref 2
/Page 4
/Encrypt 0
/ObjStm 0
/JS 0
/JavaScript 0
/AA 0
/OpenAction 0
/AcroForm 0
Extracting Text from Incremental Revisions via pdf-parser:
1
2
3
4
5
# Search for all text objects inside the PDF stream
pdf-parser --search "CONFIDENTIAL" target_confidential.pdf
# Dump raw content of object stream #12 to inspect previous revisions
pdf-parser --object 12 --raw --filter target_confidential.pdf
4. The Flawed Redaction Exploit
One of the most persistent operational blunders in government and corporate disclosures is improper redaction:
graph TD
A["User Opens PDF in Adobe Reader / Graphic Software"] --> B["Draws Black Filled Rectangle over Secret Names"]
B --> C["Clicks 'Save' (Believes text is permanently destroyed)"]
C --> D["Analyst Opens PDF in Poppler (pdftotext) or Selects All"]
D --> E["Black Box is merely a Vector Shape Layer (Z-index top)"]
E --> F["Full Plaintext Extracted Instantly from Background Layer!"]
Testing for Flawed Redactions via Command Line:
1
2
3
4
5
# Extract all raw underlying plaintext using poppler-utils
pdftotext -layout target_redacted.pdf extracted_plaintext.txt
# Inspect extracted text for previously 'blacked-out' passages
grep -i "settlement amount" extracted_plaintext.txt
Real-World Case Scenario
Case Study: Uncovering an Anonymous Whistleblower’s True Identity
An anonymous whistleblower sent a 12-page PDF dossier alleging environmental corruption by a chemical manufacturer to several major newspapers.
While the text was written from an external perspective, the recipient media group asked an OSINT analyst to verify the document’s authenticity:
- Running ExifTool revealed the
Authortag was blank, but theProducertag read:macOS Version 14.2 Quartz PDFContext.- Decompressing the PDF streams via pdf-parser located an embedded thumbnail preview image (
Obj 18).- Extracting the thumbnail revealed a full-screen screenshot of an Apple Mail inbox showing the logged-in email address of the company’s Vice President of Regulatory Affairs.
- Furthermore, on page 8, a black rectangle had been placed over an internal project code. Running
pdftotextextracted the underlying string:Project Blue-Horizon Phase 3.
Defensive Countermeasures & OpSec Recommendations
For corporate legal teams, privacy officers, and whistleblowers:
- Deploy True Redaction Tools: Never use simple highlighting or rectangle drawing tools to redact. Always use dedicated redaction features (e.g., Adobe Acrobat Pro Sanitize Document or Apply Redactions) that permanently rasterize and destroy underlying vector glyphs.
- Automate CI/CD Metadata Sanitization: Enforce an automated gateway script that executes
exiftool -all= -overwrite_originalon all public documents prior to publishing on public websites. - Convert to Clean Bitmap (Rasterization): If absolute privacy is required, convert the PDF into high-resolution PNG images (
pdftoppm -png), visually inspect the pixels, and recompile into a clean PDF (img2pdf).
Conclusion & Next Steps
Chapter 13 proves that documents are rich forensic artifacts that frequently reveal more about their creators than the text intended for publication. By mastering ExifTool, parsing PDF object streams, and testing for redaction flaws, an investigator can uncover internal organizational truths.
In Chapter 14: Photographs & Reverse Image Search, we advance to visual forensic intelligence: camera sensor PRNU fingerprinting, Error Level Analysis (ELA), and cutting-edge facial recognition engines.
Practical Document Forensics & Metadata Extraction Cheat-Sheet
| Forensic Operation | Terminal Command / Tool | Investigative Benefit |
|---|---|---|
| Comprehensive Metadata Dump | exiftool -a -u -g1 target.pdf |
Extracts author names, operating systems, printer serials, and revision timestamps |
| Extract Embedded Attachments | pdfdetach -list target.pdf && pdfdetach -saveall target.pdf |
Carves hidden binary attachments, spreadsheets, and scripts embedded in PDF objects |
| Automated Document Harvester | foca --domain target.com --extensions pdf,docx,xlsx |
Crawls target domain, extracts all public documents, and builds organizational IT profile |
| Decompress Office XML | unzip -p target.docx docProps/core.xml | xmllint --format - |
Extracts raw creator, lastModifiedBy, and revision count directly from XML headers |
| Strip Metadata for OpSec | exiftool -all= target.jpg |
Zeroes out all internal EXIF, XMP, and IPTC headers prior to public file release |
Authoritative References & Verified Investigative Sources
- Michael Bazzell, IntelTechniques: Document Analysis and Metadata Forensics. https://inteltechniques.com/book1.html.
- Phil Harvey: ExifTool Documentation and Tag Database. https://exiftool.org/.
- ElevenPaths: FOCA (Fingerprinting Organizations with Collected Archives). https://github.com/ElevenPaths/FOCA.
- NIST SP 800-88 Rev. 1: Guidelines for Media Sanitization and Metadata Purging. https://csrc.nist.gov/publications/detail/sp/800-88/rev-1/final.
- ISO 32000-1: Document Management - Portable Document Format (PDF 1.7). https://www.iso.org/standard/51502.html.
Author: Abdulrahman M. Fathi (0x3bdo) • Cyber Security Engineer & Penetration Tester