Searching leads to 503 Backend fetch failed error

[haesleinhuepf]

Hi all,

when searching for "fiji tracking" on biii.eu leads to this error at the moment:

https://biii.eu/search?search_api_fulltext=fiji+tracking

[PerrineGilloteaux]

Interestingly https://biii.eu/search?search_api_fulltext=%22Fiji+tracking%22 is working,
as well as https://biii.eu/search?search_api_fulltext=%22Fiji++Tracking%22 and https://biii.eu/search?search_api_fulltext=Fiji%2Btracking (when Fiji+tracking was entered as search in the text
https://biii.eu/search?search_api_fulltext=microtubule+tracking, to be investigated further

[PerrineGilloteaux]

The following sounds as a very similar issue , now reported by Hervé Ménager
Hi BIII friends,

It looks like there is a problem on BIII. When trying to retrieve the list of software from your server (URL http://biii.eu/soft?_format=json), the reply I get is a 503. Detailed message is:

Error 503 Backend fetch failed
Backend fetch failed

Guru Meditation:
XID: 975783143

Varnish cache server

[PerrineGilloteaux]

I am checking the host server database health and then sending an issue to our hosting server support

[PerrineGilloteaux]

I'm keeping you updated here, below is the ticket.
From You
Hosting - Error on website

Hosting concerned
web server

Domain concerned
biii.eu

Is this error present on some pages or on the whole site?
search pages e.g. http://biii.eu/soft?_format=json but working well on http://biii.eu/fiji?_format=json or https://biii.eu/search?search_api_fulltext=%22fiji+tracking%22 or

Did this error appear after a particular change?
no

Error message
Error 503 Backend fetch failed
Backend fetch failed

Guru Meditation:
XID: 975783143

Varnish cache server

Additional information
Nothing changed we think from the client side
Date received
10/11/2023 15:58

[hmenager]

Thanks @PerrineGilloteaux , just re-tested the page http://biii.eu/soft?_format=json, still getting a 503:

[PerrineGilloteaux]

Thanks @PerrineGilloteaux , just re-tested the page http://biii.eu/soft?_format=json, still getting a 503:

Yes, I ve just reported so far, so I am not expecting an improvement right now :-) , but I confirm I am getting also the error constantly on the pages I've mentionned (including the one you've reported), while some other json export (including the onbe used by comulis) are working well. Google told me it could be from the server side, so I ve contacted our host server support.

[miura]

More info: accessing from GitHub Actions fails with the same error.
Alban's test:
https://github.com/research-software-ecosystem/content/actions/runs/6955587741/job/18924654703

[oeway]

Same error happened with https://biii.eu/search?search_api_fulltext=segmentation or https://biii.eu/search?search_api_fulltext=image

But this works: https://biii.eu/search?search_api_fulltext=fiji

Could you please look into this?

[miura]

Hi @oeway

thanks for reporting the problem. My guess now is that the server running biii has low performance and the response to searches sometimes becomes too slow causing the 503 error. Do you maybe have any suggestions for testing if such a problem is happening? This does not appear in the Drupal log - I looked for any error in responding but this is not recorded. You have been running ImageJ.JS and I believe that you have good knowledge in this direction and would like to reply to you for any tips.

If this is an issue that can be solved simply by upgrading the server, we are lucky... but for this, it's better to have some clear indication that this is due to the low-performance server.

Kota

[oeway]

Hi Kota,

Thanks for looking into this! I can reliably reproduce the error with the above URL, so i think it's not a performance issue. Giving the 503 error, my guess is that the server (most likely the varnish cache server) entered a bad state.

I am not an expert in Varnish, but maybe you should check the log via "varnishlog".

[oeway]

Update: It looks like the server is getting back, I assume you did some refreshing on the caching sever since I see a very slow response on new queries but becomes rapid for a second search and onwards.
Thank you!

[oeway]

FYI: Still failing for https://biii.eu/search?search_api_fulltext=image+analysis, same 503 error.

[PerrineGilloteaux]

Hi, it works for me??

Could it be that you are searching very generic terms here so a lot of output and then time out by your own internet provider ?

Try with something like https://biii.eu/search?search_api_fulltext=%22c-elegans+analysis%22 Meaning c-elegans AND analysis (if you don't put the " " it will search for c-elegans OR analysis https://biii.eu/search?search_api_fulltext=c-elegans+analysis less relevant

[oeway]

It fails from time to time l, it I do see a pattern with more general terms fails more.

A bit of context, we are building chatbot which interface with biii, the user might trigger very general search.