More data sources: government portals, spending, corporate, and investigative databases

[JordanCoin]

Find and integrate more public data sources beyond our current 6. Focus on free, API-accessible, no-auth-required sources that help journalists and researchers follow money, power, and connections.

Tier 1 — High value, free API, easy to integrate

Federal Government Documents

• GovInfo API — Congressional reports, court opinions, Federal Register, bills, hearings. Free API key via api.data.gov. Covers all 3 branches of government.
• Federal Register API — Every federal rule, proposed rule, notice, and executive order. No API key required. JSON + CSV.
• State Department FOIA Reading Room — Searchable released documents including declassified cables and diplomatic correspondence.
• USAspending API — Every federal contract, grant, and loan since 2001. Who got the money, how much, from which agency. No auth needed. This is how you follow the money.

Corporate & Financial

• ProPublica Nonprofit Explorer API — IRS Form 990 data on every tax-exempt org in the US. Revenue, expenses, officers, compensation. Public domain.
• ProPublica Campaign Finance API — FEC data on campaign donations, PACs, independent expenditures.
• OpenSpending — 45M+ government fiscal records across 76 countries.
• Companies House (UK) — Free API, ownership, directors, filings for every UK company.

Investigative & Cross-Border

• OCCRP Aleph — 400M+ documents from 200+ datasets, 139 countries. Corporate registries, financial records, leaks, legal filings. Public access available, API for bulk search. Already uses FollowTheMoney format (we support FtM import/export!).

Tier 2 — High value, may need scraping or special access

Federal Agency Reading Rooms

Most federal agencies have electronic FOIA reading rooms with previously released documents. These don't have APIs but could be scraped:

• DOJ FOIA Library
• FTC FOIA Reading Room
• Treasury Electronic Reading Room
• FCC Electronic Reading Room
• National Archives FOIA

State & Local Data Portals

Major state open data portals (many use Socrata/CKAN with APIs):

• California Open Data
• New York Open Data
• Illinois Open Data
• Texas Open Data
• DataPortals.org — directory of 2,600+ open data portals worldwide

Specialized

• Regulations.gov API — Public comments and documents for every federal rulemaking. Follow industry lobbying on specific regulations.
• NINA (Latin America) — Collaborative database of public documents across Latin American countries. Cross-searches OpenSanctions, Aleph, ICIJ, and OpenCorporates.
• FOIA.gov API — Submit FOIA requests to any federal agency programmatically. Could wire into our request filing system.

Tier 3 — Nice to have

• Accountability Project — Curated, standardized public data (people, orgs, locations)
• GPO Federal Digital System — Full text of congressional hearings, GAO reports
• ProPublica Forensics API — Coroner/medical examiner system data by state and county

Priority

Start with USAspending (follow the money), GovInfo (follow the law), and OCCRP Aleph (follow the corruption). These three plus our existing 6 would make OpenFOIA the most comprehensive free investigative toolkit available.

Architecture note

All new sources should follow the RecordAdapter pattern in openfoia/records/. Register in __init__.py. Lazy-import any optional deps.

[JordanCoin]

Gap Analysis: What We Have vs What's Missing

Current 6 sources — what they actually answer

Source	Core question it answers	Dimension
MuckRock	"Has anyone already investigated this entity via FOIA?"	Prior investigations
DocumentCloud	"What source documents mention this entity?"	Document text
OpenCorporates	"What companies are connected to this entity?"	Corporate ownership
SEC EDGAR	"What has this public company disclosed?"	Financial filings
OpenSanctions	"Is this entity sanctioned or politically exposed?"	Risk flags
ICIJ Offshore Leaks	"Does this entity have offshore connections?"	Hidden ownership

What we CAN'T answer today

1. 💰 Follow the money — We have no spending data. We can't tell you that Palantir got $1.5B in federal contracts, or which PAC donated to which politician, or how much a nonprofit CEO makes. This is the biggest blind spot.

2. ⚖️ Follow the law — We can't see court opinions, congressional reports, or federal rulemaking. If an entity is involved in litigation or being discussed in Congress, we're blind to it.

3. 🌍 Cross-border corruption — ICIJ covers leaked papers but not the broader picture. OCCRP Aleph has 400M+ documents across 139 countries including corporate registries, financial records, and legal filings we don't touch.

4. 📋 Follow the lobbying — We can't see who's commenting on federal regulations or trying to influence rulemaking.

Priority order based on gap + ease of integration

#	Source	Gap filled	Overlap with existing?	Auth	Integration effort
1	USAspending	Federal contracts, grants, loans since 2001	Zero — completely new dimension	None	Medium — REST API, JSON
2	OCCRP Aleph	400M docs, 139 countries, corporate registries, leaks	Some (indexes OpenSanctions + ICIJ), but adds leaked docs + registries neither has	Free account	Low — already uses FollowTheMoney format, we have FtM import/export
3	ProPublica Nonprofit	IRS 990 data — revenue, expenses, officers for every US nonprofit	Zero	API key (free)	Low — simple REST
4	GovInfo	Court opinions, congressional reports, Federal Register	Slight with DocumentCloud (some court docs get uploaded)	Free key via api.data.gov	Medium — large dataset, need to scope
5	ProPublica Campaign Finance	FEC donation data, PAC spending	Zero	API key (free)	Low — simple REST
6	Federal Register	Every proposed/final rule, executive orders	Subset of GovInfo but standalone API is simpler	None	Low
7	Regulations.gov	Public comments on regulations (who's lobbying)	Zero	Free key via api.data.gov	Medium

The end state

After integrating #1-4, a journalist clicks an entity in the graph and sees:

• 🔍 Who they are — OpenCorporates, SEC filings
• 📄 What documents mention them — DocumentCloud (10M+), MuckRock FOIA requests
• 💰 How much federal money they got — USAspending contracts + grants
• 💸 Where their political money goes — ProPublica Campaign Finance
• 🏛️ What legal/congressional proceedings involve them — GovInfo
• ⚠️ Risk flags — OpenSanctions, ICIJ Offshore Leaks
• 🌍 International connections — OCCRP Aleph (400M+ docs, 139 countries)

All local. All offline-capable (after initial fetch). All purgeable. Free.

[JordanCoin]

Progress Tracker

Completed

• USAspending — 44cf95f — Federal contracts, grants, loans. No auth. Tested with Palantir ($292M DoD Maven, $276M Gotham, $252M CDAO, $195M data platform, $145M DHS ICM). Integrated into crossref.

Next up

• OCCRP Aleph — 400M docs, FtM format (we already support import/export)
• ProPublica Nonprofit — IRS 990 data, simple REST
• GovInfo — Court opinions, congressional reports, Federal Register
• ProPublica Campaign Finance — FEC donation data
• Federal Register — Rules, executive orders
• Regulations.gov — Public comments on rulemaking

Current source count: 7

muckrock, documentcloud, opencorporates, sec, opensanctions, icij, usaspending

[JordanCoin]

Update: ProPublica Nonprofit done

• USAspending — 44cf95f — Federal contracts, grants, loans
• ProPublica Nonprofit — b83e846 — IRS 990 data, 1.8M+ orgs, revenue/expenses/assets. EFF: $18M revenue, $46.7M assets. No auth needed.

Next up

• OCCRP Aleph — Requires auth (API returns 0 results without login). Deferred until we add optional API key config.
• GovInfo — Court opinions, congressional reports, Federal Register
• ProPublica Campaign Finance — FEC donation data
• Federal Register — Rules, executive orders
• Regulations.gov — Public comments on rulemaking

Current source count: 8

muckrock, documentcloud, opencorporates, sec, opensanctions, icij, usaspending, nonprofits

[JordanCoin]

Update: GovInfo done

• USAspending — 44cf95f — Federal contracts, grants, loans
• ProPublica Nonprofit — b83e846 — IRS 990 data, 1.8M+ orgs
• GovInfo — 75d1a37 — Court opinions, congressional hearings, GAO reports, Federal Register, 50+ collections. Uses DEMO_KEY (no signup). Palantir: 336 results including trademark litigation.

Next up

• OCCRP Aleph — Deferred (requires auth for API access)
• ProPublica Campaign Finance — FEC donation data
• Federal Register — Standalone API (simpler than GovInfo for rules/EOs)
• Regulations.gov — Public comments on rulemaking

Current source count: 9

muckrock, documentcloud, opencorporates, sec, opensanctions, icij, usaspending, nonprofits, govinfo

[JordanCoin]

Update: FEC + Regulations.gov done

• USAspending — 44cf95f — Federal contracts, grants, loans
• ProPublica Nonprofit — b83e846 — IRS 990 data, 1.8M+ orgs
• GovInfo — 75d1a37 — Court opinions, congressional reports, Federal Register
• FEC Campaign Finance — 4dc39be — Contribution records, PAC spending. Palantir PAC: $5K each to Wittman, Calvert, Garcia, DCCC.
• Regulations.gov — 4dc39be — Federal rulemaking documents and public comments. 3,033 results for "facial recognition."

Deferred

• OCCRP Aleph — Requires auth (API locked behind login). Will add when we implement optional API key config.
• Federal Register standalone — Redundant with GovInfo FR collection. Skip unless needed.

Final source count: 11

muckrock, documentcloud, opencorporates, sec, opensanctions, icij, usaspending, nonprofits, govinfo, fec, regulations

What a crossref on "Palantir" now returns:

• MuckRock: FOIA requests about Palantir
• DocumentCloud: 2,122 public documents mentioning Palantir
• OpenCorporates: Corporate registrations
• SEC EDGAR: Public company filings
• USAspending: $292M DoD Maven, $276M Gotham, $145M DHS ICM contracts
• ProPublica Nonprofit: Foundation/nonprofit search
• GovInfo: 336 results — trademark litigation, congressional mentions
• FEC: 54 PAC contributions to Congress members and committees
• Regulations.gov: Rulemaking documents
• OpenSanctions: Sanctions/PEP check
• ICIJ: Offshore leaks check

All free. All no-auth (or DEMO_KEY). All local after fetch.