GH-5: Write a script to find websites based off of body names

README.md

@@ -1,6 +1,6 @@
 # WebsiteScorecard
 
-CLI to scan websites from a CSV and enrich rows with check results (SSL certificate status, and more over time).
+CLI to scan websites from a CSV and enrich rows with check results (SSL certificate status, and more over time). Use the `resolve` command first when your CSV has institution names but no URLs.
 
 ## Prerequisites
 
@@ -56,7 +56,7 @@ This writes `data/mins_depts_test_scored.csv` by default (same name as input wit
 Specify an output file:
 
 ```bash
-websitescorecard scan data/mins_depts_test.csv -c URL -o data/mins_depts_scored.csv --checks ssl
+websitescorecard scan data/mins_depts_test.csv -c URL -o output/mins_depts_scored.csv --checks ssl
 ```
 
 ### 4. Check the output
@@ -79,8 +79,42 @@ Ministry,Ministry of Digital Economy,midec.gov.lk,valid,
 
 The `ssl_error` column contains the underlying error message when something went wrong. For scorecard reporting, `expired` and `invalid` can be grouped as cert problems; `unreachable` rows can be flagged separately for CSV cleanup.
 
+## Resolve names to domains
+
+If your CSV has institution or department **names** but no URLs, use `resolve` to look up official domains via web search before running `scan`. **Resolve is not a check** — it does not appear in `--checks` and is not registered alongside checks like `ssl`. Checks run against URLs you already have; resolve is a pre-scan step that adds URL column(s) from names.
+
+Typical two-step workflow:
+
+```bash
+# 1. Resolve names → domains (gov.lk and .gov only, keep up to 3 per row)
+websitescorecard resolve data/bodies.csv -c "Institution Name" \
+  --suffixes gov.lk,lk --limit 3 -o output/bodies_with_urls.csv
+
+# 2. Scan the first URL column
+websitescorecard scan output/bodies_with_urls.csv -c URL --checks ssl
+```
+
+`resolve` searches DuckDuckGo for `"{name} {query_suffix}"` (default query suffix: `Sri Lanka`), extracts hostnames from results, optionally filters by hostname suffix, and writes an enriched CSV. All original columns are preserved.
+
+**Output columns:**
+
+- `--limit 1` (default) → single column `URL` (or the name you set with `--url-column`)
+- `--limit 3` → `URL`, `url_2`, `url_3` (the first column is the primary one for `scan`)
+- `lookup_status` — `ok`, `no_results`, or `error`
+- `lookup_error` — detail when status is `error`
+
+Cells are left empty when fewer than N suffix-matching domains are found.
+
+**Skipping lookups:** If a row already has a value in the primary URL column, resolve leaves it unchanged (useful for manual overrides in a partially filled CSV).
+
+**Re-running:** Each run hits the search API again; there is no cache in v1.
+
+For a large batch (e.g. ~470 rows), use conservative settings to avoid rate limits: `--concurrency 1` (default) and `--delay 1.0` (default) imply roughly 8+ minutes minimum.
+
 ## CLI reference
 
+### `scan`
+
 ```bash
 websitescorecard scan INPUT_CSV -c COLUMN [OPTIONS]
 ```
@@ -107,6 +141,36 @@ websitescorecard scan --help
 
 At least one check must be specified via `--checks`. Running with an empty value prints an error.
 
+### `resolve`
+
+```bash
+websitescorecard resolve INPUT_CSV -c COLUMN [OPTIONS]
+```
+
+| Flag | Description |
+|------|-------------|
+| `INPUT_CSV` | Input CSV file path |
+| `-c, --column` | Column with institution/department **names** (**required**) |
+| `-o, --output` | Output path (default: `{input}_resolved.csv`) |
+| `--limit` | Max domains to keep per row (default: `1`) |
+| `--suffixes` | Comma-separated hostname suffixes to allow (e.g. `gov.lk,gov`) |
+| `--query-suffix` | Text appended to every search query (default: `Sri Lanka`) |
+| `--url-column` | Column name for the first result (default: `URL`) |
+| `--concurrency` | Parallel workers (default: `1`) |
+| `--delay` | Seconds to wait between searches per worker (default: `1.0`) |
+| `--timeout` | HTTP timeout for search requests in seconds |
+
+Examples:
+
+```bash
+# Resolve with suffix filter and multiple URLs per row
+websitescorecard resolve data/bodies.csv -c "Institution Name" \
+  --suffixes gov.lk,lk --limit 3 -o output/bodies_with_urls.csv
+
+# View all options
+websitescorecard resolve --help
+```
+
 ## Development
 
 Activate the virtual environment, then run tests:
@@ -118,6 +182,8 @@ pytest
 
 ## Adding a new check
 
+Checks are separate from domain lookup (`resolve`). To add a check that runs against URLs during `scan`:
+
 1. Create `src/websitescorecard/checks/your_check.py` implementing the `Check` protocol.
 2. Register it in `src/websitescorecard/checks/__init__.py`.
 3. Run it via `--checks` (comma-separated for multiple checks).

data/bodies.csv

@@ -0,0 +1,15 @@
+"Institution Name"
+Agriculture and Agrarian Insurance Board
+Central Engineering Consultancy Bureau and its subsidiaries and affiliated companies
+Department of Agrarian Development
+Department of Agriculture
+Department of Animal Production and Health
+Department of Cinnamon Development
+Department of Export Agriculture
+Department of Irrigation
+Department of Land Commissioner General
+Department of Land Title Settlement
+Department of Land Use Policy Planning
+Hector Kobbekaduwa Agrarian Research and Training Institute
+Institute of Surveying and Mapping
+Land Acquisition Board of Review