Align crawl controls and contributor docs

[SihanTeng]

Summary

• Make --exclude a real path-prefix match (path and descendants), matching the docs; not an arbitrary substring.
• Remove the unused --traversal flag (accepted but never read by the engine).
• Document --max-pages as an attempt cap (failed renders count).
• Add CONTRIBUTING.md and robots-block guidance for site operators (User-agent: kage).

Test plan

• urlx InScope / path-prefix tests
• go test -short -count=1 ./urlx/ ./cli/ (cli has no tests; build via ./...)

Related: #8