Separate Production and Development Dependencies

[brylie]

Objective

Separate our project's dependencies into production and development requirements to optimize deployments and clarify the project structure.

Description

Currently, all our dependencies are listed in a single requirements.txt file. This includes both production dependencies needed for the application to run and development dependencies used for testing, linting, and other development tasks. We need to separate these to ensure that only necessary dependencies are installed in production environments, potentially reducing deployment size and improving security.

Tasks

1. Create new requirements files:

   • Keep requirements.txt for production dependencies
   • Create requirements-dev.txt for development dependencies

2. Update requirements.txt:

   • Review current requirements.txt and remove any development-only dependencies
   • Keep only the dependencies required for the application to run in production

3. Create requirements-dev.txt:

   • Include -r requirements.txt at the top to include all production dependencies
   • Add development dependencies such as:
     • pytest
     • pytest-cov (if using coverage)
     • flake8 (if using for linting)
     • black (if using for formatting)
     • mypy (if using for type checking)
     • Any other development or testing libraries

4. Update development documentation:

   • Add instructions for installing development dependencies
   • Explain the difference between requirements.txt and requirements-dev.txt

5. Update CI/CD pipeline:

   • Modify GitHub Actions workflow to install requirements-dev.txt instead of requirements.txt

6. Update .gitignore:

   • Ensure it includes common Python development artifacts

7. (Optional) Add a setup.py or pyproject.toml:

   • If the project is or might become a package, consider adding one of these for more robust dependency management

Implementation Details

1. requirements.txt example:

   fastapi==0.68.0
   uvicorn==0.15.0
   # Other production dependencies...
   

2. requirements-dev.txt example:

   -r requirements.txt
   pytest==6.2.5
   pytest-cov==2.12.1
   flake8==3.9.2
   black==21.9b0
   mypy==0.910
   # Other development dependencies...
   

3. Update to GitHub Actions workflow (pytest.yml):

   - name: Install dependencies
     run: |
       python -m pip install --upgrade pip
       pip install -r requirements-dev.txt

4. Update to README.md or CONTRIBUTING.md:

   ## Development Setup
   
   1. Clone the repository
   2. Create a virtual environment: `python -m venv venv`
   3. Activate the virtual environment:
      - On Windows: `venv\Scripts\activate`
      - On macOS and Linux: `source venv/bin/activate`
   4. Install development dependencies: `pip install -r requirements-dev.txt`

Acceptance Criteria

• requirements.txt contains only production dependencies
• requirements-dev.txt includes all development dependencies and references requirements.txt
• CI/CD pipeline successfully uses requirements-dev.txt for testing
• Development documentation is updated with new setup instructions
• The application can be deployed using only requirements.txt
• Developers can set up the full development environment using requirements-dev.txt

Additional Considerations

• Consider using a tool like pip-compile to generate and manage requirements.txt and requirements-dev.txt
• Regularly review and update both requirements files to keep dependencies up-to-date
• Consider adding a Makefile or scripts to simplify common development tasks

Resources

• Python Packaging User Guide
• pip-tools documentation
• GitHub Actions documentation

Future Enhancements

• Implement dependency pinning for all packages to ensure reproducible builds
• Consider using a dependency management tool like Poetry or Pipenv for more advanced dependency management
• Set up automated dependency updates using a service like Dependabot