setup.py -gX is very slow

[jisantuc]

Problem Description

setup.py ... -gX is extremely quiet and incredibly slow. In an attempt to load all three levels of the sample data -- which is a totally reasonable thing to do that probably everyone who works on district builder will end up doing -- I've had it running for 20 minutes now and it's still going. That's not acceptable.

Replication Steps

• first, from the django container:

./setup.py config/config.xsd config/config.xml -g0 -g1 -g2

• separately: docker-compose exec django ./manage.py dbshell
• start a timer when you run: select count(1) from redistricting_geounit;
• wait thirty seconds
• run that sql command again
• observe how many records have been inserted
• note privately that there are OVER 9,000 (I have no idea how many but it is more than 9,000) records that will be inserted
• also note that the command gives you no indication that it's doing anything, and that if you didn't know where to look in the database to watch the record count increase, you'd have to fall back on checking cpu utilization in the django container to confirm that anything is going on, which is bad
• 😢

Expected Behavior or Output

Loading development data should take under twenty minutes. I think there's an easy win here -- instead of going geounit by geounit, checking what we need to do, and saving each one, we could build an array of geounits and insert them all at once. That would be risky for a file coming in from Somewhere Else in the Universe:tm:, but it's fine for a known well-formatted source.

[mcdisc]

I would be concerned about the "Somewhere Else" as Mexico City has invited a proposal to deploy DistrictBuilder for their upcoming 2020 redistricting. ============ Dr. Michael P. McDonald Associate Professor, University of Florida 352-273-2371 www.electproject.org @ElectProject

…

On Fri, Oct 20, 2017 at 10:38 AM, James Santucci ***@***.***> wrote: Problem Description setup.py ... -gX is extremely quiet and incredibly slow. In an attempt to load all three levels of the sample data -- which is a totally reasonable thing to do that probably everyone who works on district builder will end up doing -- I've had it running for 20 minutes now and it's still going. That's not acceptable. Replication Steps From the django container: - first: ./setup.py config/config.xsd config/config.xml -g0 -g1 -g2 - separately: docker-compose exec django ./manage.py dbshell - start a timer when you run: select count(1) from redistricting_geounit; - wait thirty seconds - run that sql command again - observe how many records have been inserted - note privately that there are at least 8,000 records that will be inserted - 😢 Expected Behavior or Output Loading development data should take under twenty minutes. I think there's an easy win here -- instead of going geounit by geounit, checking what we need to do, and saving each one, we could build an array of geounits and insert them all at once. That would be risky for a file coming in from Somewhere Else in the Universe™️, but it's fine for a known well-formatted source. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <PublicMapping/districtbuilder#471>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABBMSf9jwzC-rNVWusl04F1xVdqMoTv7ks5suLB_gaJpZM4QAw5R> .

[jisantuc]

I think we could handle it as a flag -- like setup.py ... -g0 -g1 -g2 --batched -- to preserve flexibility