[CDASH] set hard limit for weekly tests timeouts on Virgo
Try to fit the runtime of the weekly tests so that they can run in the main
partition on Virgo
by limiting the "obvious" too long timeouts.
These 4 places were leading by the automatic calculation to not realistic timeouts of 10000-20000 s
for single tests (the main
queue in Virgo is 8 hours
or 28800 s
for the full job of 350 tests, so too short even with 10 CPUs booked).
It is not 100% sure if this will be enough to make the automatic weekly upload to CDASH as apart from 2 or 3 tests most of them were still in the range of 1500 s
duration and would therefore not hit the new limit.
Not sure whom should review this so I pinged the main actors under.
@s.zharko Does the QA ts_eb_ideal_compare
test really need a fixed value of 10000 s
or is it a typo?
from what I can see in my logs it typically finishes in 3-10 s
even for the weekly tests
@f.kornas @f.uhlig For the JSON production test, is 40 events still reasonable?
or could it be that there is some part of the test which may conflict with other tests for disk access or such? \
- On
Virgo
I see (in the last few weeks) the test passing most of the time after~1200-1400 s
but sometimes timing out after more than6000 s
- on the
mcbmop
nodes of the control room, it similarly oscillates between2500 s
and more than6000 s
depending on the node and the week