Feat/extract website developer company

[NethmikaKekuu]

This pr closes #6

Summary

Adds a new footer_vendor check to WebsiteScorecard that scrapes the footer of government websites to identify the vendor or developer responsible for building the site.

Changes

New check — FooterVendorCheck (src/websitescorecard/checks/footer_vendor.py)

Scrapes a website's footer and extracts the vendor/developer credit using a multi-stage strategy:

1. Looks for a <footer> element, then falls back to any element with a footer-like id/class, then the bottom 20% of the page body
2. Finds <a> tags near credit trigger phrases (designed by, developed by, solution by, in association with, etc.)
3. If the link wraps only an image, extracts the domain from href instead of link text
4. Falls back to regex matching on plain footer text for un-hyperlinked vendor names
5. Scans copyright lines for external links as a last resort
   Result statuses:

Status	Meaning
found	Vendor name identified, stored in footer_vendor_error column
self_built	Site was built by the institution's own IT/ICT unit
unknown	Page loaded but no vendor credit detected
unreachable	Page could not be fetched

Scrape_URL column support (src/websitescorecard/runner.py)

Many government sites require navigating past a language selector to reach the actual content. A Scrape_URL column can now be added to the input CSV to specify the exact inner page to scrape for footer content. The check tries Scrape_URL first and falls back to the base URL if no vendor is found.

Known vendor domain mapping

Infrastructure domains are mapped to human-readable names rather than being discarded:

• gic.gov.lk → ICTA (GIC)
• icta.lk → ICTA
• slts.lk → SLT
  New data file (data/mins_depts_test_check.csv)

Input CSV with Scrape_URL populated for ministries and departments where the homepage requires a language selection step.

How to Run

websitescorecard scan data/mins_depts_test_check.csv -c URL --checks ssl,footer_vendor

Dependencies

Requires two additional packages:

pip install httpx beautifulsoup4

[gemini-code-assist]

Code Review

This pull request introduces a new FooterVendorCheck to extract website creator and vendor information from footers, registering it in the check registry and updating the runner to pass row context. It also updates the test datasets and README instructions. The reviewer feedback focuses on performance and design improvements, including: avoiding redundant network requests by de-duplicating URLs and tracking reachability during the initial fetch; reusing a single httpx.Client instance; optimizing BeautifulSoup traversal by avoiding string re-parsing and find_all(True) calls; and refactoring the runner to avoid isinstance checks by updating the base Check.run signature.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.