SolarCleanAI — Reference Implementation

Working code for the architecture in your project doc: Weather/Dust data → Feature Engineering → Model 1 (LSTM dust) → Model 2 (XGBoost power loss) → Model 3 (cleaning optimizer) → Dashboard, plus the two computer-vision "Future Extension" models built from your Kaggle links.

What's here

solarcleanai/
├── data/
│   └── data_loader.py           # Pulls/loads all 3 Kaggle sources
├── features/
│   └── feature_engineering.py   # Builds the exact 10-feature set from the architecture doc
├── models/
│   ├── dust_lstm.py             # Model 1: LSTM dust accumulation prediction
│   ├── power_loss_xgboost.py    # Model 2: XGBoost power loss prediction
│   ├── cleaning_optimizer.py    # Model 3: cleaning decision + ROI
│   ├── fault_detection_cnn.py   # Extension: VGG16 multi-class fault classifier
│   └── dust_detection_cnn.py    # Extension: InceptionV3 clean/dusty classifier
├── train_pipeline.py             # Runs all 3 core models end-to-end
└── requirements.txt

1. Setup

python -m venv venv && source venv/bin/activate   # or conda
pip install -r requirements.txt

2. Getting the 3 datasets — now fully automatic

All three real Kaggle dataset slugs were confirmed directly from the used dataset links : 1- https://www.kaggle.com/datasets/stucom/solar-energy-power-generation-dataset/data 2- https://www.kaggle.com/code/pythonafroz/cnn-vgg16-used-for-solar-panel-fault-detection/input?select=Faulty_solar_panel 3- https://www.kaggle.com/code/pythonafroz/dust-detection-on-solar-panel-using-inceptionv3 notebooks you provided (Dataset.zip — their /kaggle/input/... paths and embedded metadata datasetIds), so all three now download via kagglehub with no manual unzipping:

#	What	Kaggle dataset (owner/slug)	Subfolder
a	Weather + power generation	stucom/solar-energy-power-generation-dataset	spg.csv
b	Multi-class fault images	pythonafroz/solar-panel-images	Faulty_solar_panel/
c	Binary dust images	malkamahira/solar-panels-dirt-detection	Detect_solar_dust/

This needs a Kaggle API token: create one at kaggle.com → Account → "Create New API Token", then place kaggle.json at ~/.kaggle/kaggle.json.

python -c "from data.data_loader import get_power_data_dir, get_fault_images_dir, get_dust_images_dir; \
print(get_power_data_dir()); print(get_fault_images_dir()); print(get_dust_images_dir())"

Or just run python data/data_loader.py directly — it downloads and prints a summary of all three.

If you'd rather not re-download or want to point at data you already have locally, set the override env vars (they take priority over kagglehub):

export SOLARCLEANAI_POWER_DIR=/path/to/power_csvs
export SOLARCLEANAI_FAULT_DIR=/path/to/Faulty_solar_panel
export SOLARCLEANAI_DUST_DIR=/path/to/Detect_solar_dust

3. Column mapping — check this first

The stucom dataset's actual column names may not match feature_engineering.COLUMN_ALIASES exactly (e.g. it may not include PM10/PM2.5/AOD at all, since that's a solar plant dataset, not an air-quality dataset). After loading:

from data.data_loader import load_power_generation_data
df = load_power_generation_data()
print(df.columns.tolist())

Add any missing mappings to COLUMN_ALIASES in features/feature_engineering.py. If the dataset has no PM10/PM2.5/AOD columns at all, that's expected — either source them from a weather API (NASA POWER / ERA5, as named in your architecture diagram) or use the CNN dust_score as a proxy (see step 5).

4. Run the core 3-model pipeline (synthetic smoke test)

Runs immediately, no data download needed — proves every stage wires together, using synthetic data shaped like the real thing:

python train_pipeline.py

Once you've downloaded and column-mapped the real data, replace the make_synthetic_dataset() call in train_pipeline.main() with your real, merged DataFrame (weather + dust proxy + power_output_kw + panel_age_days + a days_since_last_cleaning column built from your actual cleaning logs via feature_engineering.add_days_since_cleaning).

5. Train the computer-vision extensions

python models/fault_detection_cnn.py   # VGG16, multi-class -- downloads pythonafroz/solar-panel-images automatically
python models/dust_detection_cnn.py    # InceptionV3, binary -- downloads malkamahira/solar-panels-dirt-detection automatically

dust_detection_cnn.get_dust_score() returns a 0–1 "how dusty is this panel" score from a single photo. Feed a batch of these into feature_engineering.ensure_all_features(df, dust_score_col='dust_score') to backfill PM10/PM2.5/AOD wherever you don't have real sensor data — this is exactly how the "Future Extension: computer vision" item in your doc plugs into the core pipeline rather than sitting separately from it. here is drive link for trained Models : https://drive.google.com/drive/u/0/folders/19LXYqxJnq2v9jkod1Swv9yZLhYouztIF

6. Evaluate results — accuracy, F1, confusion matrix, sample predictions

After training either or both CNNs, run:

python models/evaluate_cnn.py --model both

This re-creates the SAME validation split each model trained on (same validation_split=0.2, same directory), so metrics are honest — computed on images the model never saw during training. For each model it produces:

• Console output: full classification_report (precision/recall/F1 per class) + overall accuracy
• <model>_classification_report.txt — the same, saved to disk
• <model>_confusion_matrix.png — a plotted confusion matrix
• <model>_samples.png — a grid of validation images with true vs. predicted label (green title = correct, red = wrong), so you can visually spot which classes get confused (e.g. Bird-drop vs. Dusty)

Options:

python models/evaluate_cnn.py --model fault              # only the VGG16 fault model
python models/evaluate_cnn.py --model dust                # only the InceptionV3 dust model
python models/evaluate_cnn.py --n-samples 24               # bigger sample grid

7. Optimization Reports

When running the core 3-model pipeline (python train_pipeline.py), the system automatically generates presentation-ready figures and structured data files in the solar_clean_ai_outputs/ directory.

Here is the exact data measured and stored for performance tracking:

• CSV (14_day_horizon_forecast.csv): Stores the daily breakdown across the 14-day optimizer window:

  • dust_level (creeping accumulation index per day)
  • estimated_power_loss_pct (daily predicted capacity degradation)
  • Associated timeline features across the simulated horizon

• Text Report (pipeline_results_discussion.txt): A human-readable, append-only log summarizing the financial and performance improvements of the AI schedule versus standard fixed-interval cleaning:

  • Model 1 & 2 validation metrics (MSE, MAE, R²)
  • Current power loss vs. restored capacity (+21.64%)
  • Direct cleaning operational costs saved vs. a standard fixed schedule
  • Net financial saved value and System ROI rate

• High-Resolution Visuals:

  • fig1_power_improvement.png: Uncleaned power loss vs. AI scheduled clean restoration.
  • fig2_cleaning_cost_reduction.png: Fixed 7-day schedule costs vs. SolarCleanAI optimized costs.
  • fig3_revenue_and_roi.png: Unmitigated cumulative revenue loss vs. AI mitigation threshold.

8. Where CNN models save

• CNNs save to whatever save_path you pass train()

Notes on scaling to a real farm

• panel_age_days / days_since_last_cleaning should come from your asset registry and maintenance logs, not synthetic data — that's the main manual-integration step beyond what's here.
• CostAssumptions in cleaning_optimizer.py is where you plug real labor/water/electricity rates for Egypt-specific ROI numbers.
• The optimizer currently evaluates a fixed horizon with a simple recursive dust-escalation assumption; swapping it for genuine per-day weather forecasts (rather than "last known + drift") is the most impactful next accuracy improvement.