Skip to content

README.md

Latest commit

 

History

History
308 lines (239 loc) · 9.86 KB

README.md

File metadata and controls

308 lines (239 loc) · 9.86 KB
Raw

🇦🇪 UAE Public Data Engine

CI License: MIT Python 3.11 Code style: ruff

A high-performance Python FastAPI engine for standardizing UAE Open Data (Real Estate & Infrastructure) into consistent, developer-friendly APIs.

🎯 Vision & Impact

The UAE is rapidly becoming a global leader in smart city innovation and data-driven governance. However, accessing and integrating public data from various government sources remains fragmented and technically challenging for developers.

This project bridges that gap by providing a standardized, high-performance API layer that transforms raw UAE open data into consistent, well-documented JSON schemas.

Strategic Alignment

  • UAE National Strategy for AI 2031: Supports the national vision of becoming a world leader in AI by making government data more accessible for AI/ML applications
  • Dubai Open Data Strategy: Aligns with Dubai's initiative to make government data available to businesses and researchers
  • Market Transparency: Enables real estate professionals, investors, and researchers to access standardized property transaction data
  • Developer Ecosystem: Lowers the barrier for PropTech startups and developers building UAE-focused applications

Who Benefits

  • Real Estate Professionals: Access standardized transaction data for market analysis
  • PropTech Startups: Build applications on consistent, well-documented APIs
  • Researchers & Analysts: Access normalized datasets for urban planning and economic research
  • AI/ML Engineers: Obtain clean, structured data for training models on UAE real estate patterns

⚡ Features

  • 9 DLD Datasets: Transactions, Rents, Projects, Valuations, Land, Buildings, Units, Brokers, Developers
  • Standardized Schemas: Consistent JSON responses across all data sources
  • Aggregation Endpoints: Group by area, property type, date ranges
  • Summary Endpoints: Market overview, price trends, developer performance
  • Matching Endpoints: Cross-dataset joins for enriched views
  • FastAPI: Modern, high-performance async web framework
  • Dockerized: Production-ready containerization with multi-stage builds
  • Poetry: Modern dependency management and virtual environments
  • Linting & Formatting: Pre-configured Ruff for code quality
  • CI/CD: Automated testing and linting via GitHub Actions
  • OpenAPI Documentation: Auto-generated interactive API docs at /docs

🛠 Tech Stack

Component Technology
Engine Python 3.11 + FastAPI
Data Models Pydantic v2
Data Loading CSV to in-memory
Architecture Microservice (Dockerized)
Data Sources Dubai Land Department (DLD)
CI/CD GitHub Actions

Data Sources

Source Data Type Status
Dubai Land Department (DLD) Real Estate Transactions, Rents, Projects ✅ Integrated
Dubai Pulse Infrastructure, Transportation, Environment 🚧 Planned
Abu Dhabi DMT Transport & Mobility 📋 Planned

🚀 Getting Started

Prerequisites

  • Python 3.11+
  • Poetry
  • Docker & Docker Compose (optional, for containerized run)

Installation

  1. Clone the repository:

    git clone https://github.com/Dynamic-Web-Lab/uae-public-data-engine.git
cd uae-public-data-engine

  2. Install dependencies:

    make install
# OR
poetry install

Development

Start the development server with hot-reload:

make run
# App will be running at http://localhost:8000
# Interactive docs at http://localhost:8000/docs
# ReDoc at http://localhost:8000/redoc

Run linter and tests:

make check

Docker

# Build and run with Docker Compose
docker-compose up --build

# Or build manually
docker build -t uae-public-data-engine .
docker run -p 8000:80 uae-public-data-engine

📊 API Endpoints

Data Endpoints

Method Endpoint Description
GET /api/v1/transactions/ List transactions with filters
GET /api/v1/transactions/{id} Get transaction by ID
GET /api/v1/rents/ List rental contracts
GET /api/v1/projects/ List development projects
GET /api/v1/valuations/ List property valuations
GET /api/v1/land/ List land parcels
GET /api/v1/buildings/ List buildings
GET /api/v1/units/ List individual units
GET /api/v1/brokers/ List licensed brokers
GET /api/v1/developers/ List registered developers

Aggregation Endpoints

Method Endpoint Description
GET /api/v1/aggregation/transactions/by-area Transaction metrics by area
GET /api/v1/aggregation/transactions/by-date Transaction trends over time
GET /api/v1/aggregation/rents/by-area Rental metrics by area
GET /api/v1/aggregation/projects/by-status Projects by completion status
GET /api/v1/aggregation/projects/by-developer Projects per developer

Summary Endpoints

Method Endpoint Description
GET /api/v1/summary/market-overview High-level market snapshot
GET /api/v1/summary/price-trends Price per sqm trends

Matching Endpoints

Method Endpoint Description
GET /api/v1/matching/transaction/{id}/details Full transaction with context
GET /api/v1/matching/area/{code}/profile Complete area profile
GET /api/v1/matching/broker/{id}/transactions Broker transaction history
GET /api/v1/matching/search Unified search across datasets

Sample Response

{
  "status": "success",
  "source": "dubai_land_department",
  "metadata": {
    "total_items": 20,
    "filters_applied": {
      "area": "Dubai Marina"
    }
  },
  "data": [
    {
      "transaction_number": "TN-2024-001234",
      "transaction_date": "2024-01-15",
      "transaction_type": "Sale",
      "area": "Dubai Marina",
      "property_sub_type": "Apartment",
      "amount": 1850000.00,
      "property_size_sqm": 116.00,
      "rooms": 2
    }
  ],
  "pagination": {
    "page": 1,
    "page_size": 50,
    "total_pages": 1,
    "total_items": 20
  }
}

📂 Project Structure

.
├── app/
│   ├── api/
│   │   └── v1/
│   │       ├── router.py              # Main v1 router
│   │       └── endpoints/             # API endpoints
│   │           ├── transactions.py
│   │           ├── rents.py
│   │           ├── projects.py
│   │           ├── aggregation.py
│   │           ├── summary.py
│   │           └── matching.py
│   ├── schemas/
│   │   ├── common.py                  # Shared models
│   │   └── dld/                       # DLD Pydantic models
│   ├── services/
│   │   ├── data_loader.py             # CSV data loading
│   │   ├── aggregation.py             # Aggregation logic
│   │   └── matching.py                # Data joining logic
│   ├── core/
│   │   └── config.py
│   └── main.py
├── data/
│   ├── dld/                           # DLD CSV data files
│   │   ├── transactions/
│   │   ├── rents/
│   │   ├── projects/
│   │   └── ...
│   └── dubai-pulse/                   # Future data sources
├── tests/
├── .github/
├── pyproject.toml
├── Dockerfile
├── docker-compose.yml
└── Makefile

🗺 Roadmap

Phase 1: Foundation (Current)

  • Project scaffolding with FastAPI
  • Docker configuration
  • CI/CD pipeline
  • Basic health check endpoint
  • Pydantic models for 9 DLD datasets
  • Sample CSV data files
  • Basic CRUD endpoints
  • Aggregation endpoints
  • Summary endpoints
  • Matching/join endpoints

Phase 2: Data Integration

  • Dubai Pulse API integration
  • Pydantic models for infrastructure data
  • Schema validation and transformation

Phase 3: Production Features

  • Rate limiting
  • Authentication
  • Caching layer
  • Monitoring and metrics

Phase 4: Advanced Analytics

  • Time series analysis
  • Predictive analytics
  • Market forecasting

🤝 Contributing

We welcome contributions! This project especially needs help with:

  • New Data Sources: Mapping Abu Dhabi DMT schemas, Sharjah data
  • Data Models: Expanding Pydantic models for different property types
  • Tests: Improving test coverage
  • Documentation: API usage examples, tutorials

Please read the CONTRIBUTING.md and CODE_OF_CONDUCT.md for details.

Good First Issues

  • Add schema for Abu Dhabi property transactions
  • Create data model for commercial real estate
  • Add unit tests for schema validation
  • Improve error handling for API rate limits

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Built with ❤️ for the UAE developer community

Report Bug · Request Feature