V2.0.0

.github/workflows/python-app.yml

@@ -15,7 +15,7 @@ jobs:
 
     strategy:
       matrix:
-        python: ["3.9", "3.10", "3.11", "3.12", "3.13"]
+        python: ["3.11", "3.12", "3.13"]
     runs-on: ubuntu-latest
 
     steps:

.pre-commit-config.yaml

@@ -0,0 +1,49 @@
+# Pre-commit hooks configuration for gitlabber
+# Install with: pip install pre-commit && pre-commit install
+
+repos:
+  # General file checks
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files
+      - id: check-json
+      - id: check-toml
+      - id: check-merge-conflict
+      - id: debug-statements
+      - id: mixed-line-ending
+
+  # Python code formatting with black
+  - repo: https://github.com/psf/black
+    rev: 24.2.0
+    hooks:
+      - id: black
+        language_version: python3
+        args: ['--line-length=100']
+
+  # Python linting with ruff (fast, modern replacement for flake8)
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.2.2
+    hooks:
+      - id: ruff
+        args: [--fix, --exit-non-zero-on-fix]
+
+  # Type checking with mypy
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.8.0
+    hooks:
+      - id: mypy
+        additional_dependencies: [types-PyYAML, types-all]
+        args: [--ignore-missing-imports, --no-strict-optional]
+        exclude: ^tests/
+
+  # Import sorting with isort (configured to be compatible with black)
+  - repo: https://github.com/pycqa/isort
+    rev: 5.13.2
+    hooks:
+      - id: isort
+        args: ["--profile", "black", "--line-length", "100"]
+

CHANGELOG.md

@@ -1,6 +1,67 @@
 # Changelog
 
 <!--next-version-placeholder-->
+## [Unreleased]
+
+## [2.0.0] - 2025-01-XX
+
+### Added
+- **Major Performance Feature**: Add `--api-concurrency` option for parallel API calls during tree building. This dramatically speeds up tree discovery for large GitLab instances with many groups and subgroups. Real-world performance improvements: **4-6x speedup** (e.g., 96s → 16-21s for instances with 21+ subgroups). The feature includes:
+  - Parallel group processing at the top level
+  - Parallel subgroup detail fetching (batch processing)
+  - Parallel subgroups and projects fetching within each group
+  - Automatic connection pool sizing to prevent urllib3 warnings
+  - Thread-safe rate limiting to respect GitLab API limits
+  - Configurable via `--api-concurrency N` (default: 5, range: 1-20) or `GITLABBER_API_CONCURRENCY` environment variable
+  - Optional `--api-rate-limit` to set custom rate limits (default: 2000 requests/hour)
+- **Enhanced Progress Reporting**: Progress bars now show estimated time remaining (ETA) and current operation details (cloning, pulling, fetching, processing)
+- **Actionable Error Messages**: Error messages now include context-specific suggestions with actionable steps and links to documentation
+- **Pydantic-based Configuration**: Configuration management with automatic validation and environment variable support
+- **Environment Variable Support**: All configuration options can now be set via environment variables (e.g., `GITLABBER_API_CONCURRENCY`, `GITLABBER_TOKEN`)
+- **Comprehensive Documentation**: Added module-level docstrings, API documentation, `DEVELOPMENT.md` with architecture docs, and enhanced `CONTRIBUTING.md`
+- **Pre-commit Hooks**: Added pre-commit hooks with black, ruff, mypy, and isort for code quality
+- **Test Utilities**: Added comprehensive test helpers and utilities for better test organization
+- **Performance Tests**: Added performance benchmarks and e2e tests for API concurrency
+- **Custom Exception Hierarchy**: Structured exception classes for better error handling
+
+### Changed
+- **BREAKING**: Require Python 3.11 or newer (dropped Python 3.9 and 3.10 support)
+- **BREAKING**: Migrate CLI implementation from argparse to Typer for modern option parsing and help output
+- **BREAKING**: Replace tqdm-based progress bars with Rich for improved CLI UX (different visual appearance)
+- Convert CLI enums to `enum.StrEnum` for clearer string semantics
+- Modernize type hints throughout codebase (`list[str]` instead of `List[str]`)
+- Convert `GitAction` to `@dataclass` for better code clarity
+- Use `pathlib.Path` consistently throughout codebase
+- Refactor `GitlabTree` into smaller, focused components:
+  - `GitlabTreeBuilder`: Builds tree structure
+  - `TreeFilter`: Handles filtering logic (functional approach)
+  - `UrlBuilder`: Centralized URL construction
+- Extract git operations into separate classes:
+  - `GitRepository`: Wraps git operations for a single repo
+  - `GitActionCollector`: Collects git actions
+  - `GitSyncManager`: Manages concurrent git operations
+- Improve tree filtering with functional approach and predicate composition
+- Enhance error handling with specific exceptions and better context
+- Improve input validation with `urllib.parse` for URLs
+- Update dependencies: anytree 2.13.0, GitPython 3.1.45, python-gitlab 7.0.0, PyYAML 6.0.3
+- Automatically configure HTTP connection pool size based on `--api-concurrency` to prevent connection pool warnings
+- Improve test coverage from 92% to 97%
+- Standardize logging (use `log.critical()` instead of `log.fatal()`)
+- Use f-strings consistently throughout codebase
+
+### Removed
+- Remove unused `typing` dependency (built-in since Python 3.5+)
+- Remove unused `docopt` dependency
+- Remove refactoring-related comments from codebase
+- Remove unused enum argparse methods (handled by Typer)
+
+### Fixed
+- Fix error handling to provide actionable suggestions
+- Fix progress reporting to show accurate ETA
+- Fix connection pool warnings with dynamic sizing
+- Fix test coverage gaps in error handling and edge cases
+
+
 ## [1.2.8] - 25/3/2025
 ### Added
 - Add support for shared projects fetching

CONTRIBUTING.md

@@ -1,37 +1,225 @@
-Contributing
-When contributing to this repository, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository before making a change.
+# Contributing to Gitlabber
 
-Please note we have a code of conduct, please follow it in all your interactions with the project.
+Thank you for your interest in contributing to Gitlabber! This document provides guidelines and instructions for contributing.
 
+## Code of Conduct
 
-Dependencies
-============
-* pyvenv
-* pytest
-* pytest-cov
-* pytest-integration
+Please note we have a [Code of Conduct](CODE_OF_CONDUCT.md). Please follow it in all your interactions with the project.
 
+## Getting Started
 
-Setup
-=====
-* Environment
-```
-python3 -m venv .pyvenv
-source ./.pyvenv/bin/activate
-pip install pytest pytest-cov pytest-integration wheel
-python -m build
-```
+### Prerequisites
 
-* Run Tests
-```
+- Python 3.11 or higher
+- Git 2.0 or higher
+- pip
+
+### Development Setup
+
+1. **Fork and clone the repository:**
+   ```bash
+   git clone https://github.com/ezbz/gitlabber.git
+   cd gitlabber
+   ```
+
+2. **Create a virtual environment:**
+   ```bash
+   python3 -m venv .venv
+   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+   ```
+
+3. **Install dependencies:**
+   ```bash
+   pip install --upgrade pip
+   pip install -e ".[test]"
+   ```
+
+   This installs the package in editable mode with all test dependencies.
+
+4. **Verify installation:**
+   ```bash
+   gitlabber --version
+   pytest --version
+   ```
+
+## Development Workflow
+
+1. **Create a branch:**
+   ```bash
+   git checkout -b feature/your-feature-name
+   # or
+   git checkout -b fix/your-bug-fix
+   ```
+
+2. **Make your changes:**
+   - Follow the code style guidelines (see below)
+   - Write or update tests
+   - Update documentation as needed
+
+3. **Run tests:**
+   ```bash
+   pytest
+   ```
+
+4. **Check code quality:**
+   ```bash
+   # Run linters (if configured)
+   ruff check .
+   mypy gitlabber/
+   ```
+
+5. **Commit your changes:**
+   ```bash
+   git add .
+   git commit -m "feat: add new feature"
+   ```
+
+   Use conventional commit messages:
+   - `feat:` for new features
+   - `fix:` for bug fixes
+   - `docs:` for documentation changes
+   - `test:` for test changes
+   - `refactor:` for code refactoring
+   - `chore:` for maintenance tasks
+
+6. **Push and create a Pull Request:**
+   ```bash
+   git push origin feature/your-feature-name
+   ```
+
+## Code Style
+
+- **Python Version:** Python 3.11+ (use modern Python features)
+- **Type Hints:** Use type hints for all function signatures
+- **Docstrings:** Follow Google-style docstrings for all public APIs
+- **Formatting:** Code should be formatted with `black` (if configured)
+- **Imports:** Use absolute imports, group by standard library, third-party, local
+- **Naming:**
+  - Classes: `PascalCase`
+  - Functions/variables: `snake_case`
+  - Constants: `UPPER_SNAKE_CASE`
+
+## Testing
+
+### Running Tests
+
+```bash
+# Run all tests
 pytest
+
+# Run with coverage
+pytest --cov=gitlabber --cov-report=html
+
+# Run specific test file
+pytest tests/test_git.py
+
+# Run with verbose output
+pytest -v
+
+# Run only fast tests (skip integration tests)
+pytest -m "not integration_test"
 ```
 
-* Release
+### Writing Tests
+
+- Place tests in the `tests/` directory
+- Test files should be named `test_*.py`
+- Use descriptive test function names: `test_<what>_<condition>_<expected_result>`
+- Use fixtures from `conftest.py` for common test setup
+- Use test helpers from `tests/test_helpers.py` for reusable utilities
+- Mock external dependencies (GitLab API, Git operations)
+- Aim for high test coverage (>90%)
+
+### Test Structure
+
+```python
+def test_function_name_condition_expected():
+    """Test description."""
+    # Arrange
+    # Act
+    # Assert
 ```
-pip install --upgrade pip
+
+## Pull Request Process
+
+1. **Before submitting:**
+   - Ensure all tests pass
+   - Update documentation if needed
+   - Add changelog entry if applicable
+   - Ensure code follows style guidelines
+
+2. **PR Description:**
+   - Clearly describe what changes were made
+   - Explain why the changes were needed
+   - Reference any related issues
+   - Include screenshots if UI changes
+
+3. **Review process:**
+   - Maintainers will review your PR
+   - Address any feedback or requested changes
+   - Keep PRs focused and reasonably sized
+
+## Building and Releasing
+
+### Building
+
+```bash
 pip install build
 python -m build
+```
+
+This creates distribution packages in the `dist/` directory.
+
+### Testing Distribution
+
+```bash
+# Check the built package
 twine check dist/*
-twine upload dist/*
-```
+
+# Test installation
+pip install dist/gitlabber-*.whl
+```
+
+### Release Process
+
+Releases are handled by maintainers. The process includes:
+1. Update version in `pyproject.toml` and `gitlabber/__init__.py`
+2. Update `CHANGELOG.md`
+3. Create a git tag
+4. Build and upload to PyPI
+
+## Getting Help
+
+- **Issues:** Open an issue for bugs or feature requests
+- **Discussions:** Use GitHub Discussions for questions
+- **Email:** Contact maintainers via email if needed
+
+## Dependencies
+
+### Runtime Dependencies
+
+See `pyproject.toml` for the complete list. Main dependencies:
+- `anytree` - Tree data structure
+- `globre` - Glob pattern matching
+- `pyyaml` - YAML parsing
+- `pydantic` - Configuration validation
+- `typer` - CLI framework
+- `rich` - Progress bars and formatting
+- `GitPython` - Git operations
+- `python-gitlab` - GitLab API client
+
+### Development Dependencies
+
+- `pytest` - Testing framework
+- `pytest-cov` - Coverage reporting
+- `pytest-integration` - Integration test support
+- `coverage` - Code coverage analysis
+
+## Questions?
+
+If you have questions about contributing, feel free to:
+- Open an issue
+- Start a discussion
+- Contact the maintainers
+
+Thank you for contributing to Gitlabber!