Add community health files and modern Python best practices

[Copilot]

Implements comprehensive project governance, security, and contributor infrastructure for the setlr repository.

Community Health Files

• CONTRIBUTING.md - Development setup with ./script/bootstrap, PEP 8/257 standards, nose2 testing, commit conventions, PR process, pre-commit hooks
• CODE_OF_CONDUCT.md - Contributor Covenant v2.1, enforcement contact: mccusj@cs.rpi.edu
• SECURITY.md - Supported versions (1.0.x), 48-hour response SLA, vulnerability reporting process, security best practices

GitHub Templates

• Pull Request Template - Structured checklist for type of change, testing performed, documentation updates, CHANGELOG entries
• Bug Report Form (.yml) - Required fields for environment (SETLr/Python/OS versions), reproduction steps, sample data
• Feature Request Form (.yml) - Problem/solution descriptions, use case analysis, feature area classification, priority levels

Dependency Management

• Dependabot - Weekly updates on Mondays for pip and GitHub Actions, grouped by dev/prod dependencies, 5 PR limit
• requirements.txt - Production dependencies extracted from pyproject.toml (18 packages)
• requirements-dev.txt - Testing (nose2, coverage), linting (flake8, pylint, vulture), build tools (build, wheel, twine)

Documentation

Created missing advanced topics referenced in README but previously absent:

• docs/advanced.md - Multi-source transforms, conditional loading, performance optimization (streaming, batching), CI/CD integration patterns
• docs/sparql.md - SPARQL queries as data sources, authenticated endpoints, loading to update endpoints, Python API examples
• docs/shacl.md - Validation constraints (cardinality, datatype, patterns), validation reports, pyshacl integration

CI/CD Enhancements

• release.yml - Automated releases on v* tags: extracts version-specific changelog, creates GitHub release, builds distributions, publishes to PyPI
• build.yml - Added twine check for package validation on Python 3.11

Code Quality

• pre-commit-config.yaml - Black (24.1.1), Flake8 (using setup.cfg), isort (black profile), standard hooks (trailing-whitespace, end-of-file-fixer, check-yaml, check-merge-conflict)

README Updates

• Added codecov badge placeholder
• Linked CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md in respective sections

Post-Merge Setup Required

1. Add PYPI_API_TOKEN secret to repository settings
2. Configure Codecov integration
3. Run pre-commit install in local development environments

Original prompt

Add Community Health Files and Modern Best Practices

Overview

Implement best practices to improve project governance, security, and contributor experience for the setlr repository.

Files to Create

1. Community Health Files

CONTRIBUTING.md

Create a comprehensive contributing guide that includes:

• Development setup instructions (using existing ./script/bootstrap)
• Code standards (PEP 8, docstrings)
• Testing guidelines (using nose2)
• Commit message conventions
• Pull request process
• Reference to existing development scripts in script/ directory

CODE_OF_CONDUCT.md

Create a Code of Conduct based on Contributor Covenant v2.1:

• Include pledge and standards
• Specify enforcement contact: mccusj@cs.rpi.edu
• Provide clear examples of acceptable/unacceptable behavior

SECURITY.md

Create a security policy that includes:

• Supported versions table (1.0.x supported)
• Vulnerability reporting process (email: mccusj@cs.rpi.edu)
• Security best practices for using SETLr
• 48-hour response time commitment

2. GitHub Templates

.github/PULL_REQUEST_TEMPLATE.md

Create a PR template with sections for:

• Description
• Motivation and context (with issue linking)
• Type of change (bug fix, feature, breaking change, docs)
• Testing performed
• Checklist (code style, documentation, tests, CHANGELOG)

.github/ISSUE_TEMPLATE/bug_report.yml

Create a structured bug report template with fields for:

• Bug description
• Steps to reproduce
• Expected behavior
• Environment (SETLr version, Python version, OS)
• Additional context

.github/ISSUE_TEMPLATE/feature_request.yml

Create a feature request template with fields for:

• Problem description
• Proposed solution
• Alternatives considered
• Additional context

3. Dependency Management

.github/dependabot.yml

Configure Dependabot for:

• Python pip dependencies (weekly updates on Mondays)
• GitHub Actions dependencies (weekly updates)
• Grouped updates for development and production dependencies
• Labels: "dependencies" and "automated"
• Open PR limit: 5

requirements.txt

Extract production dependencies from pyproject.toml:

rdflib>=6.0.0
pandas>=0.23.0
jinja2
numpy
cython
requests
toposort
beautifulsoup4
lxml
six
xlrd
ijson
click
tqdm
requests-testadapter
python-slugify
pyshacl[js]
future

requirements-dev.txt

Create development dependencies file:

-r requirements.txt

# Testing
nose2>=0.9.0
coverage>=5.0

# Linting
flake8>=3.8.0
pycodestyle>=2.6.0
pylint>=2.6.0
vulture>=2.0

# Build tools
build>=0.7.0
wheel>=0.36.0
twine>=3.4.0

4. Documentation Enhancement

docs/advanced.md

Create the missing advanced features documentation that is referenced in docs/README.md but doesn't exist:

• Overview of advanced capabilities
• Multi-source transforms
• Conditional loading
• Performance optimization (streaming, batch processing)
• Error handling and debugging
• Integration patterns (CI/CD, Docker)
• Links to existing advanced docs (streaming-xml.md, python-functions.md, sparql.md, shacl.md)

5. CI/CD Enhancements

.github/workflows/build.yml

Create a build and publish workflow:

• Trigger on push to main/develop and tags (v*)
• Build distribution packages using python -m build
• Check packages with twine
• Upload artifacts
• Auto-publish to PyPI on version tags (using PYPI_API_TOKEN secret)

.github/workflows/release.yml

Create a release workflow:

• Trigger on version tags (v*)
• Extract version-specific changelog entries from CHANGELOG.md
• Create GitHub release with changelog as release notes

6. Code Quality Tools

.pre-commit-config.yaml

Configure pre-commit hooks:

• Pre-commit-hooks: trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files, check-merge-conflict
• Black formatter (version 24.1.1)
• Flake8 linter (using existing setup.cfg configuration)
• isort (with black profile)

Update CONTRIBUTING.md to include pre-commit setup instructions.

7. Updated README.md

Add to the README.md file:

• Code coverage badge placeholder (for future Codecov integration)
• Link to CONTRIBUTING.md in the Contributing section
• Link to CODE_OF_CONDUCT.md
• Link to SECURITY.md for security reports

Implementation Notes

Compatibility

• Maintain compatibility with existing workflows (.github/workflows/test.yml and lint.yml)
• Preserve existing setup.cfg, pyproject.toml, and unittest.cfg configurations
• Keep existing script/bootstrap, script/build, script/release functionality
• Don't modify core Python code

Style Consistency

• Match existing documentation style and formatting
• Use existing badge format from README.md
• Follow existing CHANGELOG.md format
• Maintain Apache License 2.0 references

Testing

• Ensure all new YAML files are valid
• Verify Markdown formatting
• Check that dependabot.yml syntax is correct
• Validate GitHub Actions workflow syntax

Success Criteria

• All community health files created and properly formatted
• Depend...

This pull request was created from Copilot chat.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.