pyaqsapi generating incorrect bdate parameter in bystate.monitors function, and runs slower than RAQSAPI

[DEQrporter]

Hey EPA,

I have noticed that the pyaqsapi monitor service, bystate.monitors(), does not return the same results as a similar call from RAQSAPI, an API call to the web or an AMP500 report.

This is the query for pyaqsapi: df = pd.DataFrame(aqs.bystate.monitors("88101", datetime.date(1950, 1, 1), datetime.date(2024, 12, 31), "41"))
This is the query for the web: https://aqs.epa.gov/data/api/monitors/byState?email=my_email&key=my_key&param=88101&bdate=19500101&edate=20241231&state=41

I have attached the results from both and an AMP500 report. I couldn't specify dates in the AMP500 report, so I queried back to 1950 through the web and pyaqsapi. I only have checked for the prescence of a specific site, 410310133. It also takes at least three minutes for pyaqsapi to complete the process and it also just seems to repeat the first 21 items.

Can you verify this is a bug and fix it? I'm happy to help out with more troubleshooting or diagnostics.

Thanks!

Ryan Porter
Oregon DEQ

pyaqsapi_monitors_1950.csv

AMP500_monitors.csv
AMP500_monitors.txt
APM500_monitors.pdf

API_monitors_1950.json

[vipinkataria2209]

I would like to contribute and will work on this issue.

[mccroweyclinton-EPA]

Thanks, I am sorry it took so long to respond. I will look into this.

[mccroweyclinton-EPA]

Good morning, again sorry I took so long to respond, I must have missed the notification that someone submitted a bug report. I was forwarded your email from the AQS team since this is not their project. For future reference please do not message the AQS regarding issues with pyaqsapi or RAQSAPI.

Anyways to answer your question:

Please note that in your AMP500 report on the pdf, the first page you have under "SELECTED OPTIONS"
[*] SITE MONITOR TRANSACTIONS = SITE BASIC & MONITOR BASIC
This means that the AMP500 report that you ran will return basic site information as well as monitor information, the pyaqsapi.bystate.monitors() function in pyaqsapi and the aqs_monitors_by_state() function in RAQSAPI will only return monitor information and not include site information.

I just ran that query in both RAQSAPI and pyaqsapi and both packages generated the same url (with the exception of the bdate parameter which is a problem in pyaqsapi)

library(lubridate)
library(RAQSAPI)
mntrs <- aqs_monitors_by_state(parameter="88101", bdate=ymd("1950-1-1"), edate=ymd("2024-12-31"), stateFIPS="41", return_header=TRUE)
mntrs[[1]]$Header$url
[1] "https://aqs.epa.gov/data/api/monitors/byState?email=<redacted>&key=<redacted>&param=88101&bdate=19500101&edate=20241231&state=41"

from datetime import date
import pyaqsapi as aqs
mntrs = aqs.bystate.monitors(parameter="88101", bdate=date(year=1950, month=1, day=1), edate=date(year=2024, month=12, day=31), stateFIPS="41", return_header=True)
mntrs[1].get_url()
Out[24]: 'https://aqs.epa.gov/data/api/monitors/byState?email=<redacted>&key=<redacted>&param=88101&bdate=20240101&edate=20241231&state=41'

There does appear to be something wrong with how the dates are being generated in pyaqsapi however as you can see the bdate in python should be 19500101 and 19501231 for mntrs[1].get_url(). Also the query did take a lot longer to run in python than it did in R so I will look further into that too.

I will assume that the data that is returned from both pyaqsapi and RAQSAPI would be the same if the bdate parameter in pyaqsapi were generated correctly since neither package alter the data other than comining multiple dataframes into a single large one and possibly renaming of columns in RAQSAPI which I have not implemented in pyaqsapi yet.

[DEQrporter]

Okay. Thanks for picking this up!

[mccroweyclinton-EPA]

I would like to contribute and will work on this issue.

@vipinkataria2209 do you have a lead on what is causing this issue?

[mccroweyclinton-EPA]

@DEQrporter well it is easy to see why your call in pyaqsapi take so much longer than it does in RAQSAPI, you are requesting 74 years of data. I have never requested that many years of data in a single call, during all of my testing. In fact, it is probably better to not overload the AQS DataMart API server with a request spanning such a large time frame in a single call.

Anyways, to address the issue as hand, pyaqsapi has a function __aqs_ratelimit() that is called after each API call to the server; each year of requested data will require 1 API call. By default __aqs__ratelimit makes a 5 second call to time.sleep() as a simple pause between API calls. So 74 years * 5 second delay/year results in a 370 second delay added by the ratelimit. Also 370 seconds ~ 6.16 minutes. So the equivalent call in pyaqsapi adds at least 6.16 minutes of delay (minus the throttling action in RAQSAPI). This delay also does not account for the time that it takes for the API server to return data which in itself can take quite some time. The rate limit in RAQSAPI is managed differently because the R package:httr2 has internal functions that manage rate limits differently. It seems that I may have to implement the ratelimit differently in pyaqsapi so that API calls are on par with that of what would be achievable in RAQSAPI, hopefully it can be done in a way that is similar to how it is in R by the httr2::req_throttle function.

I am still working on the other, more important issue of this function returning incorrect data.

[mccroweyclinton-EPA]

I have created a separate issue, #28 to address improving pyaqsapi's throttling (rate limit). This issue is left open to address pyaqsapi returning incorrect results.

[DEQrporter]

Yeah, I only queried to 1950 to show all the 88101 monitors I get from pyaqsapi and AQS AMP report, since I couldn't specify a date range in AQS. Thanks for continuing to look into this. I'll poke around some and try to see what's going on.

[mccroweyclinton-EPA]

@DEQrporter, after I couldn't figure out why this call was returning incorrect dates, I took a break from pyaqsapi to work on improving RAQSAPI. For the past few days I've been taking a second look at this issue and could not recreate the issue of pyaqsapi returning incorrect dates. It could be due to the fact that, recently I have completely deleted and rebuilt my python virtual environment that I was working in. Were you ever able to recreate the issue that I had where running this didn't return data from the correct date range?

 aqs.bystate.monitors(parameter="88101", bdate=date(year=1950, month=1, day=1), edate=date(year=2024, month=12, day=31), stateFIPS="41", return_header=True)

because I can't seem to recreate that issue. If not then maybe i'll just blame my venv =). Anyways I am working on a fix to this and #28, at least for the monitors call. I expect to push the fix in the next few days.

[mccroweyclinton-EPA]

@DEQrporter, please test out 7461299 and let me know if this fixes this issue for you, at least for the monitors function it should be much faster. The rest of the function should still work the same.

[DEQrporter]

@mccroweyclinton-EPA, tried it out. It processed much more quickly and I didn't get repeated rows like I did before. So, something worked! Thanks!