Demo: Benchmark comparing powerwalk vs os.walk for package discovery

[Peter554]

A small demo showing off https://github.com/Peter554/powerwalk, for your interest/consideration @seddonym

Results from my machine:

❯ uv run benchmark_walk.py
Benchmarking package discovery
Package: octoenergy
Directory: /Users/peter.byfield/projects/kraken-core/src/octoenergy

Running warm-up and verification...
os.walk found: 65752 modules
powerwalk found: 65761 modules

WARNING: 9 modules only found by powerwalk:
  octoenergy.plugins.territories.aus.markets.electricity.meter_data_quality.interfaces.supportsite.templates.partials
  octoenergy.plugins.territories.aus.markets.gas.interfaces.supportsite.templates.support.workflows
  octoenergy.plugins.territories.aus.markets.gas.interfaces.supportsite.templates.support.workflows.partials
  octoenergy.plugins.territories.gbr.interfaces.supportsite.templates.electricity.meter-points.partials
  octoenergy.plugins.territories.nld.markets.common.interfaces.templates.support.billing.popups
  octoenergy.plugins.territories.nzl.markets.electricity.interfaces.supportsite.templates.support.industry.nzl.electricity_service_orders
  octoenergy.plugins.territories.usa.markets.ercot.interfaces.supportsite.templates.support.accounts.popovers
  octoenergy.plugins.territories.usa.markets.ercot.interfaces.supportsite.templates.support.meter_points
  octoenergy.plugins.territories.usa.markets.ercot.interfaces.supportsite.templates.support.meter_points.pop_ups

============================================================
Running 10 iterations each...

os.walk run 1: 0.6553s
os.walk run 2: 0.7320s
os.walk run 3: 0.6544s
os.walk run 4: 0.7183s
os.walk run 5: 0.6345s
os.walk run 6: 0.7121s
os.walk run 7: 0.6500s
os.walk run 8: 0.6373s
os.walk run 9: 0.6999s
os.walk run 10: 0.6314s

powerwalk run 1: 0.4323s
powerwalk run 2: 0.3870s
powerwalk run 3: 0.4518s
powerwalk run 4: 0.3668s
powerwalk run 5: 0.4795s
powerwalk run 6: 0.3750s
powerwalk run 7: 0.4738s
powerwalk run 8: 0.3699s
powerwalk run 9: 0.4691s
powerwalk run 10: 0.4032s

============================================================
Results:
============================================================

os.walk:
  Average: 0.6725s
  Min:     0.6314s
  Max:     0.7320s

powerwalk:
  Average: 0.4208s
  Min:     0.3668s
  Max:     0.4795s

Speedup: 1.60x
✓ powerwalk is 1.60x faster

[codspeed-hq]

CodSpeed Performance Report

Merging #262 will not alter performance

_{Comparing Peter554:speedywalk-demo (af5c383) with main (558e62c)}

Summary

✅ 26 untouched
⏩ 23 skipped¹

Footnotes

1. 23 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[seddonym]

Thanks for sharing!

I like the look of powerwalk, nice idea. I tried switching to the PowerWalkModuleFinder and trying it out building a large package locally - it does seem to be a bit faster, but it's somewhat marginal - maybe 3.2s -> 2.9s. I wonder what's going on there.

I'd be open to including something like this in theory, though given only the slight improvement it might be better to concentrate on the other PR...what do you think?

[Peter554]

Thanks for sharing!

I like the look of powerwalk, nice idea. I tried switching to the PowerWalkModuleFinder and trying it out building a large package locally - it does seem to be a bit faster, but it's somewhat marginal - maybe 3.2s -> 2.9s. I wonder what's going on there.

I'd be open to including something like this in theory, though given only the slight improvement it might be better to concentrate on the other PR...what do you think?

Thanks for taking a look. Yes I think likely the other PR is a better direction. But this one was still fun for me at least.

[Peter554]

Closing this PR since I think #263 is more useful