Investigate why HB is slow

[radmilak]

Investigate the infamous "general slowness" in HB.
What we know:

• It does not happen constantly - seems to go in "waves" that last several hours
• Effects all tools, though larger ProTo-tools or tools where there is a lot of computation between gui reloads are more effected due to bigger chance for ProTo to timeout (default 30s)
• Seems at least sometimes to also effect the general HB gui (clicking in the menu, loading a tool before any input is given)
• There might be a connection to what happens on the server - on one occasion we knew there were big jobs running when HB was very slow, pausing the jobs did not help though
• All instances are equally effected (dev, sandbox, /hb)
• One of the theories is that there is race condition/resource blocking happening somewhere, perhaps on the shelves or in genome-related resources, though on one occasion GSuite parsing was timed and identified as taking very long time and we didn't find it access any shared resources
• Another theory is that this is somehow related to Galaxy/how jobs are started
• There is nothing exceptional found in logs except ProTo timeouts when slowness occurs
• Sveinung: It is probable it could be something with the server and not software
• Sveinung: It could be related to the forking of the process in ProTo

[sveinugu]

Could also be (and perhaps most probable) something to do with the server setup, and not software related.

[sveinugu]

@radmilak Any updates on the insilico monitoring software you queried about?

[radmilak]

@sveinugu Some small updates - I found out I don't really have permissions to check the system logs on insilico so didn't find out anything by myself. But I contacted hpc-drift about this yesterday, haven't gotten any answer yet. Will post update here when they answer.

[radmilak]

@sveinugu @morj-uio So, hpc-drift has restarted the monitoring client so it works again. They didn't say anything about why it was down but I guess if it would crash again we can ask them to investigate more then. It looks like this:

The monitoring pages are only accessible with an IP from the UiO range, so to do this from home you have to either tunnel via an UiO server (I also got this browser plugin so only grafana.uio.no requests use the tunnel) or set up a VPN but I didn't find much info about this on UiO website, so haven't tried. Or it should probably work with the UiO remote desktop thing.
To access it when you have the right IP, go to https://grafana.uio.no/ , click Home->host-resources and choose insilico.

[sveinugu]

Created Epic issue for this: https://app.zenhub.com/workspaces/elixir-oslo-5e2fa59480ef93e1d919fc6d/issues/hyperbrowser/genomic-hyperbrowser/134