currently, both extractors run at 7 am UTC (9 am Israel) time and as the git hub actions shows, branch extractor runs smoothly while the information extractor crashes due to SQL time out.
Need to look for a way to query all the rows in the table branch price in a reasonable time so that It will be possible to not only update the entries in the table but do it quickly.
As of right now, the moment the extractors run, all rows from the relevant table are queried and converted into a dict in order to allow for fast searching. This is an alternative to the first method which was in each iteration query an item to see it is in the table already, or if its price is updated. The reason I went with this is to force the bottleneck of the code to be at one stage of the code instead of having a recurring bottleneck (query each row individually ).
currently, both extractors run at 7 am UTC (9 am Israel) time and as the git hub actions shows, branch extractor runs smoothly while the information extractor crashes due to SQL time out.
Need to look for a way to query all the rows in the table branch price in a reasonable time so that It will be possible to not only update the entries in the table but do it quickly.
As of right now, the moment the extractors run, all rows from the relevant table are queried and converted into a dict in order to allow for fast searching. This is an alternative to the first method which was in each iteration query an item to see it is in the table already, or if its price is updated. The reason I went with this is to force the bottleneck of the code to be at one stage of the code instead of having a recurring bottleneck (query each row individually ).