Requesting new option to avoid hitting `daily_captures_limit`

[barkoder]

$ curl -s -H 'Accept: application/json' -H 'Authorization: LOW AUTH:KEY' https://web.archive.org/save/status/user
{"available":8,"daily_captures":10199,"daily_captures_limit":10000,"processing":0}

Wayback machine has been slowly reducing the available page captures per user. It used to be 100,000 then 80,000, then 40,000. Now it's a mere 10,000 .

I've been hitting that limit every day these days.

So I'm requesting a new option.

If daily_captures_limit - daily_captures is below a certain specified number, then

• user specifiable suboption A: sleep job until next UTC day begins.

OR

• user specifiable suboption B: safely break out of the job immediately.

I need this feature because there are other important URLs that I need to reserve some captures for, and I don't want to hit the daily_captures_limit.

Thanks!

[bac0id]

Hi. As an alternative, you can sign up for additional Internet Archive accounts for additional captures.

[barkoder]

About 24 hours after I opened this issue, the available captures returned back to 40000 .

Hi. As an alternative, you can sign up for additional Internet Archive accounts for additional captures.

I could, but I don't want to burden(abuse) their free and wonderful service. If they've reduced the number of available captures per user on their end, I'll assume IA servers are overloaded in some capacity.
I honestly don't mind waiting it out.

I need this feature specifically because I archive ~2000-4000 "endangered" URLs each day. I unequivocally need ~4000 slots to be free(reserved) on my account for these URLs.

I don't want spn.sh eating up all the available capture slots while I'm archiving non-endangered URLs. So whilst I'm spn-ing the non-endangered URLs, if daily_captures_limit - daily_captures < 4000 , then I want spn.sh to sleep and wait till the beginning of the next UTC day to continue capturing.

[bac0id]

spn.sh can resume an aborted session. I think a new option to restrict the number of captures per session can be added. So spn.sh can pause itself, and then you can restart spn.sh as a scheduled job with cron. But this approach only works for a single device, not for multiple devices using the same IA authentication.

If you want to pause spn.sh by daily_captures, this means to have extra communication with IA to get latest daily_captures, which may reduce the speed of sending capture tasks. I feel the IA server limits the communication frequency to approximately once every 2 seconds, as I frequently encounter SSL errors during communication.

[barkoder]

I've not encountered SSL errors but IA does sometimes return nothing when I check daily_captures

I have to while loop it until I get an actual result.

while true ; do curl -s -H 'Accept: application/json' -H 'Authorization: LOW S3:PASS' https://web.archive.org/save/status/user && break ; sleep 10 ; done

I personally don't mind the penalty induced(if any) in sending capture tasks.
But you may have the user configure the polling frequency depending on their needs.

If the user wants a more aggressive(more accurate) capture slot reservation, and the user is okay with any task capturing speed penalty(if any) associated with it, then the user may set it to LOW number – like having spn.sh do daily_captures_limit - daily_captures once every 10s , let's say.
But if user doesn't mind losing a few more capture slots(less strict, less accurate), and/or the user is NOT okay with any associated task capturing speed penalty, then the user may set the polling frequency to HIGH number – like once every 10m.

Let the user set whether spn.sh should check free slots and do the arithmetic every 10 seconds or 10 minutes.

No cron please.

Thanks!

[bac0id]

If the user wants a more aggressive(more accurate) capture slot reservation, and the user is okay with any task capturing speed penalty(if any) associated with it, then the user may set it to LOW number – like having spn.sh do daily_captures_limit - daily_captures once every 10s , let's say.

A good idea. Will this feature be added? @overcast07

By the way, I opened another issue #42 for SSL errors from IA. I did some research on it.

[overcast07]

I'm willing to merge pull requests, so it could happen that way, but as you can see from the changelog I haven't written any code for this script in almost 2 years, so I can't guarantee that I would ever implement it myself. I agree that adding a feature to spread out the allotted captures evenly throughout the day would be helpful, though. Sorry if this is a disappointing answer.

[user-1-2-3-4-5]

In addition, for "anonymous" users the limit at the time is just 200 URLs a day. I haven't mass-saved webpages to the Wayback Machine in over a year and somehow remembered that the script doesn't support IA accounts.

You have reached your daily not-logged-in captures limit of 200 webpages, log in to archive up to 30,000 webpages per day.

---

I need this feature specifically because I archive ~2000-4000 "endangered" URLs each day. I unequivocally need ~4000 slots to be free(reserved) on my account for these URLs.

@barkoder What kind of pages are you archiving, if you don't mind me asking? Are you saving complete threads (each page with ~25 replies) on a discussion board or something similar?

[overcast07]

the script doesn't support IA accounts

No, that is not correct. You can authenticate by using your account's S3 keys. The script doesn't support cookie-based authentication, though.