[Improvement] Move dashboard creation on upload into background task

[starshine-bcit]

We have deployed the dashboard server internally, and folks are very happy with the initial results. However, we have run into an issue when uploading the outputs.

We are using a custom database implementation and are feeding A LOT of data into it. It takes a little bit to query all the results needed to generate the dashboard. Granted, our implementation could use some optimization.

When uploading a result via the API, the call is blocked until the result is parsed, the data inserted, a get all query is executed, and the new dashboard is generated. This is leading to very long apparent upload times, when in reality it's blocked on the get all query. This is compounded by multiple outputs being uploaded in series by our tooling.

Our workaround for now has been to implement an in-memory cache of the data (appended on each upload), but this is less than ideal.

It would be ideal if the API call only blocks on the parsing and insert of the new data, so that it could return a success or not. Then, perhaps a background task is triggered which will update the dashboard some configurable timeout (such that with x seconds of no uploads, the new dashboard is created, where x >= 0).

What do you think of this improvement?

Unfortunately, we are not authorized to make external contributions, but this may change in the future.

[timdegroot1996]

Hi @starshine-bcit,

Thanks for reporting this issue that you are experiencing.
It's very cool to hear you're using a custom database implementation. I made that but haven't had much feedback, so I'm glad people are enjoying it!

Regarding your issue, I get your problem so I'll look at ways to optimize this usage.
I am thinking along the lines of removing the auto-update of the admin page on every upload/delete.
It might be a good idea to just add a "refresh admin page data" button or something along those lines, after which the get call is executed.

Would that be a good solution for your usecase?

Tim

[starshine-bcit]

Hi @timdegroot1996

To clarify, the problem is not with the admin page data, but rather the data used to generate the dashboard html page. For example, the logs from one upload query show the life cycle:

 2. Processing output XML(s)
  Processing output XML 'output_Test-Run.xml'
Finished creating database runs in 0.59 seconds
  Processed output XML 'output_Test-Run.xml' in 1.4 seconds
======================================================================================
 5. Creating dashboard HTML
Using cached database data
  created dashboard '/app/robot_dashboard.html' in 0.79 seconds
INFO:     10.132.16.110:53580 - "POST /add-output-file HTTP/1.1" 200 OK

What I proposed above is to return 200 after the line starting with "Processed output XML ...`, then move the other two tasks into the background.

The problem task is when it is creating the dashboard HTML and querying everything from the database. As you can see, we have added a line indicating it's using the cached data, but if it wasn't, that is where the long-running query executes.

robotframework-dashboard/robotframework_dashboard/robotdashboard.py

Lines 215 to 223 in 23742bd

	def create_dashboard(self):
	"""Function that creates the dashboard HTML"""
	console = ""
	if self.generate_dashboard:
	start = time()
	console += self._print_console(f" 5. Creating dashboard HTML")
	self.database.open_database()
	dashboard_data = self.database.get_data()
	self.database.close_database()

That said, I agree that the improvement to the admin page would be useful as well.

[timdegroot1996]

@starshine-bcit I think I get what you're saying now. Thanks for clarifying. The problem lies with the regeneration fo the dashboard which is causing the long load times.

Would it then not make more sense to still work towards the solution I proposed but implement it differently?
So instead of making a button that "refreshes the admin page", I would (also?) make a button to "refresh the dashboard" only on button press instead of on each upload?

The idea is the same but this would be easier to implement and makes usage a bit more clear.
Your suggestion regarding a wait before you start updating the dashboard also makes some sense but I feel that would complicate the code and usage.

What do you think?

[starshine-bcit]

@timdegroot1996 thank you. Your solution makes sense to us, and we agree it's simpler.

In terms of implementation, it would be great if the "refresh the dashboard" button was fairly obvious. The use case here is non-technical users who may not be overly familiar with the application.

Similarly, it would be ideal if there was an option to always generate the dashboard at application startup, regardless of whether or not automatic regeneration is happening. Our implementation is running in k8s without persistent storage, so we are currently relying on the initial generation at server startup. Of course, we could add a persistent volume, but it's simpler to keep everything but the database ephemeral.