read_wateruse development idea

[elmeraa-usgs]

I took a gander at the new read_wateruse() fxn that you and Cee worked on in this MR. It's really nice to have this functionality added to dR, so thank you for those additions.

I wanted to flag one thing for the national/full-CONUS use case, in case it's useful as you think about more development.

Earlier, I was pulling all four irrigation variables for CONUS at HUC12 resolution to make national maps (wu-irrigation-wd: irrwdtot, irrwdgw, irrwdsw; wu-irrigation-cu: irrcutot).

Because read_wateruse() takes one location per call and pages at 500 HUC12s, assembling CONUS means looping the 18 HUC2 regions (and for CONUS that is~83,000 HUC12s so roughly 166 paginated requests per variable, I believe). Even after batching all of a model's variables into a single sweep, that's ~330 requests total, and end-to-end it took me over 20 minutes (and I hit an occasional 180s request timeout along the way).

Here's a reproducible example of read_wateruse() for CONUS irrigation withdrawals:

library(dataRetrieval)
library(purrr)

conus_huc2 <- sprintf("%02d", 1:18)  # CONUS HUC2 regions

system.time({
  irr_wd_total <- map(conus_huc2, \(h)
                read_wateruse(
                  model = "wu-irrigation-wd",
                  variable = c("irrwdtot"),
                  location = paste0("huc2:", h),
                  timeres = "monthly",
                  startdate = "2020-01",
                  enddate = "2020-12"
                )
  ) |> list_rbind()
})

For comparison, the approach @aaarcher-usgs and I used previously, pulling the pre-made wide csvs straight from the NWDC file directory, was faster: it's a single file request per variable (the whole CONUS record, already year_month × HUC12), no pagination.

base_url <- "https://water.usgs.gov/nwaa-data/data-file-directory?path="
csv_path <- paste0(
  "data/water-use/wu-irrigation-wd/irrwdtot/",
  "irrwdtot_wu-irrigation-wd_CONUS_200001-202012_wide.csv"
)

system.time({
  httr2::request(paste0(base_url, csv_path)) |>
    httr2::req_perform(path = "irrwdtot_conus_wide.csv")  
})

irr_wd_total <- readr::read_csv("irrwdtot_conus_wide.csv")

So for national pulls the file-directory route is one request vs. hundreds. This isn't a blocker at all, read_wateruse() is clearlyfully functional works well for subsets of data. But this issue may propose a few ideas for the national-scale case such as:

• Fewer requests for whole-country pulls since the runtime is driven by paginating ~83k HUC12s. Something like a CONUS/"all" location shortcut, an option to hand back the pre-built wide CONUS files for whole-country requests, or a maybe a higher page limit would help (?).
• An option to return wide (year_month × HUC12) instead of long. This would save the large pivot needed to join to HUC12 geometries for mapping (one column per HUC12). It'd also shrink the payload a bit (long repeats the huc12_id/year_month keys on every row).

These are just a couple of my thoughts after having used it the new function for ongoing national wateruse viz work.