Downloading stash for Standard - see debug info

[OldGothith]

Using Procurement 1.30.1
After loading 31 stash tabs I receive an error below for, "The remote server returned an error: (429) Too Many Requests"?
There has not been any throttle while loading the 31 stash tabs.
Attaching full DebugInfo.log at bottom.

Preview of the error:
[26-11-2023 19:57] Failed to build HTTP request and get response for: method=GET, url='https://www.pathofexile.com/character-window/get-stash-items?league=Standard&tabs=1&tabIndex=30&accountName=Gothith&realm=pc', allowAutoRedirects=, requestData='': The remote server returned an error: (429) Too Many Requests.
[26-11-2023 19:57] Error downloading stash for Standard, exception details: System.Net.WebException: The remote server returned an error: (429) Too Many Requests.
at System.Net.HttpWebRequest.GetResponse()
at POEApi.Transport.HttpTransport.BuildHttpRequestAndGetResponse(HttpMethod method, String url, Nullable1 allowAutoRedirects, String requestData) at POEApi.Transport.HttpTransport.PerformHttpRequest(HttpMethod method, String url, Nullable1 allowAutoRedirects, String requestData)
at POEApi.Transport.HttpTransport.GetStash(Int32 index, String league, String accountName, String realm, Boolean refresh)
at POEApi.Transport.HttpTransport.GetStash(Int32 index, String league, String accountName, String realm)
at POEApi.Transport.CachedTransport.GetStash(Int32 index, String league, String accountName, String realm, Boolean refresh)
at POEApi.Model.POEModel.GetStash(Int32 index, String league, String accountName, String realm, Boolean forceRefresh)
at POEApi.Model.POEModel.GetAllTabs(String league, String accountName, Stash stash, String realm)
at POEApi.Model.POEModel.GetStash(String league, String accountName, String realm)
[26-11-2023 19:57] System.Exception: Downloading stash for Standard, details logged to DebugInfo.log, please open a ticket at https://github.com/Stickymaddness/Procurement/issues and include your DebugInfo.log
at POEApi.Model.POEModel.GetStash(String league, String accountName, String realm)
at Procurement.ViewModel.LoginWindowViewModel.LoadStashItems(Character character)
at Procurement.ViewModel.LoginWindowViewModel.d__24.MoveNext()
at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Procurement.ViewModel.LoginWindowViewModel.<>c__DisplayClass23_0.b__0()
at System.Threading.Tasks.Task.Execute()

Looking forward to this application to work/support!
DebugInfo.log

[thailyn]

Thanks for the report. Are you consistently getting this error, or only sometimes? Are you using other programs that access the PoE API at the same time?

[OldGothith]

Thanks for the report. Are you consistently getting this error, or only sometimes? Are you using other programs that access the PoE API at the same time?

Same result every time yes, and no other programs installed for the API.
Restarted windows 11 after new installation more than once with the same result.
Note: most of my tabs are in folders if that has an effect.
Thank you for responding!
Let me know if you need more information. :)

[thailyn]

Thanks for the additional details. I am also able to replicate this issue. GGG probably changed how the rate limiting works, or at least the limits. I'll take a look.

[thailyn]

GGG added (oh, boy, a while ago) additional headers to API responses with information about rate limits that Procurement does not utilize. Details here (that does not talk about any recent changes, if any, but it does document the format).

Example values from a request I sent:

Access-Control-Expose-Headers: X-Rate-Limit-Policy, X-Rate-Limit-Rules, X-Rate-Limit-Account, X-Rate-Limit-Account-State, X-Rate-Limit-Ip, X-Rate-Limit-Ip-State
X-Rate-Limit-Policy: backend-item-request-limit
X-Rate-Limit-Rules: Account,Ip
X-Rate-Limit-Account: 30:60:60,100:1800:600
X-Rate-Limit-Account-State: 1:60:0,34:1800:0
X-Rate-Limit-Ip: 45:60:120,180:1800:600
X-Rate-Limit-Ip-State: 1:60:0,34:1800:0

Based on these headers, the current rate limits for accessing stash tabs (when using a session id) appears to be 30 requests per 60 seconds, and 100 requests per 1800 seconds (30 minutes). The request there per 60 seconds is lower than what Procurement expects (42 requests/60 seconds), so it will consistently bump into the rate limit. It does not take into account any additional rate limits, so the 100 requests per 30 minutes (if that is indeed correct) might drive some other changes in Procurement.

[OldGothith]

Knowing the primary issue sounds promising.
Without GGG providing the utility for searching our stashes, many exiles are lost searching hundreds of tabs (mostly remove only) for gear/gems/jewels et al.
God speed sir

[thailyn]

Doing some internal testing: setting the API request limit in Procurement to 30/60 seconds works to avoid the first limit, but, as expected, does not avoid the second. However, while manually inspecting the headers in the responses to the throttled requests, the X-Rate-Limit-Account and X-Rate-Limit-Account-State did not have the expected values. Namely, they were something like (I neglected to copy the exact values):

X-Rate-Limit-Account: 10:60:60,50:1800:600
X-Rate-Limit-Account-State: 1:60:0,1:1800:0

That is, the rate limits were reduced and the account state was reset (including implying no throttling was active, as the third term for each of the limits was 0), but the requests were still being rejected with 429 errors.

Investigation ongoing....

[OldGothith]

Hi Chuck, Let me know if you need me to beta test. I have several hundred tabs, so if it works for me you should be good to go! Regards, Brian

…

________________________________ From: Charles Macanka ***@***.***> Sent: Wednesday, December 6, 2023 10:11 PM To: Procurement-PoE/Procurement ***@***.***> Cc: OldGothith ***@***.***>; Author ***@***.***> Subject: Re: [Procurement-PoE/Procurement] Downloading stash for Standard - see debug info (Issue #1158) Doing some internal testing: setting the API request limit in Procurement to 30/60 seconds works to avoid the first limit, but, as expected, does not avoid the second. However, while manually inspecting the headers in the responses to the throttled requests, the X-Rate-Limit-Account and X-Rate-Limit-Account-State did not have the expected values. Namely, they were something like (I neglected to copy the exact values): X-Rate-Limit-Account: 10:60:60,50:1800:600 X-Rate-Limit-Account-State: 1:60:0,1:1800:0 That is, the rate limits were reduced and the account state was reset (including implying no throttling was active, as the third term for each of the limits was 0), but the requests were still being rejected with 429 errors. Investigation ongoing.... — Reply to this email directly, view it on GitHub<#1158 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BEIWV36CQSRJIBTO3UGOK43YIFMXVAVCNFSM6AAAAAA73OLAQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBUGY3TMNRQHA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[godkill2]

Any way to disable procurement from trying to load up standard characters?