Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
indent_style = space
indent_size = 2

[*.py]
indent_size = 4

[Makefile]
indent_style = tab
16 changes: 1 addition & 15 deletions .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -1,15 +1 @@
# These are supported funding model platforms

github: 508-dev # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
patreon: # Replace with a single Patreon username
open_collective: # Replace with a single Open Collective username
ko_fi: # Replace with a single Ko-fi username
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
liberapay: # Replace with a single Liberapay username
issuehunt: # Replace with a single IssueHunt username
lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
polar: # Replace with a single Polar username
buy_me_a_coffee: # Replace with a single Buy Me a Coffee username
thanks_dev: # Replace with a single thanks.dev username
custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']
github: 508-dev
29 changes: 29 additions & 0 deletions .github/ISSUE_TEMPLATE/docs_request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: Documentation Request
description: Request new or improved documentation.
title: "docs: "
labels:
- documentation
body:
- type: textarea
id: gap
attributes:
label: Documentation Gap
description: What is missing, unclear, stale, or hard to find?
validations:
required: true
- type: textarea
id: audience
attributes:
label: Audience
description: Who needs this documentation?
placeholder: API consumers, scraper operators, contributors, agents, etc.
validations:
required: false
- type: textarea
id: suggested-location
attributes:
label: Suggested Location
description: Where should this live?
placeholder: README.md, docs/USAGE.md, docs/ARCHITECTURE.md, CONTRIBUTING.md
validations:
required: false
41 changes: 41 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Feature Request
description: Propose a new scraper capability or workflow.
title: "feature: "
labels:
- enhancement
body:
- type: textarea
id: problem
attributes:
label: Problem
description: What user, scraper, or downstream-consumer problem should this solve?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposed Solution
description: What should change?
validations:
required: true
- type: textarea
id: output-contract
attributes:
label: Output Contract
description: Describe any expected JSON shape, field additions, or compatibility constraints.
validations:
required: false
- type: textarea
id: alternatives
attributes:
label: Alternatives Considered
description: What other approaches are available?
validations:
required: false
- type: textarea
id: context
attributes:
label: Additional Context
description: Links, sanitized payloads, screenshots, examples, constraints, or non-goals.
validations:
required: false
22 changes: 14 additions & 8 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,21 @@
<!--- Provide a general summary of your changes in the Title above. -->
## Summary

<!--- Optionally prefix with feat/chore/fix: -->
-

## Description
## Scraper Impact

<!--- Describe your changes in detail. What component does this change affect? -->
- [ ] Saved-list extraction
- [ ] Place-page extraction
- [ ] CLI or public API
- [ ] Documentation/tooling only

## Related Issue
## Validation

<!--- If there are any related issues or Notion tasks, please link here, or delete this section or write N/A. -->
- [ ] Ran the narrowest relevant checks.
- [ ] Added or updated focused tests for parser, scraper, CLI, or public API changes.
- [ ] Updated docs or examples when behavior changed.

## How Has This Been Tested?
## Risk

<!--- Please describe how you tested or plan to test your changes. -->
- [ ] Low risk, isolated change.
- [ ] Needs reviewer attention for schema drift, browser behavior, LLM repair, dependency changes, or output-contract compatibility.
22 changes: 21 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,22 @@ on:
push:
branches:
- main
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

permissions:
contents: read

jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false

- name: Set up Python
uses: actions/setup-python@v6
Expand All @@ -19,6 +29,8 @@ jobs:

- name: Install uv
uses: astral-sh/setup-uv@v7
with:
enable-cache: true

- name: Sync dependencies
run: uv sync --locked --dev
Expand All @@ -30,6 +42,8 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false

- name: Set up Python
uses: actions/setup-python@v6
Expand All @@ -38,6 +52,8 @@ jobs:

- name: Install uv
uses: astral-sh/setup-uv@v7
with:
enable-cache: true

- name: Sync dependencies
run: uv sync --locked --dev
Expand All @@ -49,6 +65,8 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false

- name: Set up Python
uses: actions/setup-python@v6
Expand All @@ -57,9 +75,11 @@ jobs:

- name: Install uv
uses: astral-sh/setup-uv@v7
with:
enable-cache: true

- name: Sync dependencies
run: uv sync --locked --dev

- name: Test
run: uv run python -m unittest discover -s tests
run: ./scripts/test.sh
9 changes: 8 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,16 @@ Run the local gates before opening or updating a PR:
```bash
./scripts/lint.sh
./scripts/typecheck.sh
uv run python -m unittest discover -s tests
./scripts/test.sh
./scripts/check-all.sh
```

`./scripts/check-all.sh` is the same local gate in one command. It runs linting,
type checking, and unit tests.

Dependency changes should follow [docs/supply-chain.md](docs/supply-chain.md):
keep `uv.lock` committed and prefer locked installs.

## Pull Request Expectations

- Keep changes scoped to the scraper behavior being fixed.
Expand Down
12 changes: 11 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,14 +196,24 @@ the CLI, when a refresh only needs overview facts.
LLM setup, cache behavior, and downstream refresh examples
- [Architecture](docs/ARCHITECTURE.md): extraction layers, diagnostics, LLM repair,
translation memory, and session/concurrency design
- [Supply Chain](docs/supply-chain.md): dependency cooldowns, locked installs,
and local artifact handling
- [Contributing](CONTRIBUTING.md): local development, tests, PR expectations,
and translation-memory promotion
- [Security](SECURITY.md): vulnerability reporting, secret handling, and
dependency policy

## Development

```bash
uv sync --dev
./scripts/lint.sh
./scripts/typecheck.sh
uv run python -m unittest discover -s tests
./scripts/test.sh
```

Run the complete local gate with:

```bash
./scripts/check-all.sh
```
31 changes: 31 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Security Policy

## Reporting Vulnerabilities

Do not open public issues for vulnerabilities or leaked secrets.

Report security concerns to the project maintainers through the repository's
private security advisory flow or another private channel configured by the
maintainers.

## Secret Handling

- Keep secrets in environment variables or local ignored files.
- Never commit real `.env` files, tokens, private keys, credentials, Google Maps
session state, browser profiles, or production data.
- Use documented examples for configuration only.
- Keep debug dumps, screenshots, and browser artifacts out of committed docs
unless they are intentionally sanitized fixtures.

## Dependency Policy

This project uses `uv` with dependency cooldowns and locked installs:

- `pyproject.toml` sets `exclude-newer = "7 days"` for `uv` and `uv pip`.
- CI should use `uv sync --locked --dev`.
- Dependency changes should keep `uv.lock` committed and reviewable.

## GitHub Actions

Workflows should use least-privilege permissions, pinned or reviewed action
versions, and `persist-credentials: false` where practical.
33 changes: 33 additions & 0 deletions docs/supply-chain.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Dependency Supply-Chain Policy

This project is a Python package managed with `uv`.

## uv

Dependency resolution uses a seven-day cooldown:

```toml
[tool.uv]
exclude-newer = "7 days"

[tool.uv.pip]
exclude-newer = "7 days"
```

Keep `uv.lock` committed. CI and release workflows should install with:

```bash
uv sync --locked --dev
```

## Runtime Dependencies

Treat dependency changes as executable-code changes. Before adding or upgrading
packages, review the package purpose, release recency, lockfile diff, and any
native or browser automation surface area.

## Local Artifacts

Do not commit browser profiles, session directories, raw Google Maps payload
dumps, LLM cache files, learned translation-memory files, or screenshots unless
they are intentionally sanitized fixtures.
4 changes: 4 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ dev = [
[tool.uv]
exclude-newer = "7 days"

[tool.uv.pip]
exclude-newer = "7 days"

[tool.hatch.build.targets.wheel]
packages = ["src/gmaps_scraper"]

Expand All @@ -38,6 +41,7 @@ strict = true
[tool.ruff]
line-length = 100
target-version = "py314"
src = ["src", "tests"]

[tool.ruff.lint]
select = ["B", "E", "F", "I", "UP"]
Expand Down
6 changes: 6 additions & 0 deletions scripts/check-all.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -euo pipefail

./scripts/lint.sh
./scripts/typecheck.sh
./scripts/test.sh
4 changes: 4 additions & 0 deletions scripts/test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -euo pipefail

uv run python -m unittest discover -s tests