Data Aggregation
Need
-Scrapers for all locations
-resource list of possible dropoff site lists
-research
Scrapers
-run regularly, update API
-report of what changes were made, logged where?
-maybe build in delays if too many calls to their servers
-data cleaning duplicates!
-limit to NY state
-list of NY zip codes
-DCA pop up sites, April 1 first available
API
-every location
-columns (lat/long, hours, address, type of establishment)
-contact info, phone, website, email, etc
-easy to add new scrapers
-up to date
-give source of data (url)
-provide closest sites for entered address
Language telling people to contact that source to update the data
Pre-March 11
-Set up GitHub -done
-Compile resource list -done
-NY zipcodes -done
-Create issues for scraper and building API -discusssed
-Database host? -discussed, likely use borrowed AWS
-how many read/writes?
-find free hosting service
-whatever the cap is for storage, be under it
-Tag in Zach from AG’s Office
Data Aggregation
Need
-Scrapers for all locations
-resource list of possible dropoff site lists
-research
Scrapers
-run regularly, update API
-report of what changes were made, logged where?
-maybe build in delays if too many calls to their servers
-data cleaning duplicates!
-limit to NY state
-list of NY zip codes
-DCA pop up sites, April 1 first available
API
-every location
-columns (lat/long, hours, address, type of establishment)
-contact info, phone, website, email, etc
-easy to add new scrapers
-up to date
-give source of data (url)
-provide closest sites for entered address
Language telling people to contact that source to update the data
Pre-March 11
-Set up GitHub -done
-Compile resource list -done
-NY zipcodes -done
-Create issues for scraper and building API -discusssed
-Database host? -discussed, likely use borrowed AWS
-how many read/writes?
-find free hosting service
-whatever the cap is for storage, be under it
-Tag in Zach from AG’s Office