TSAN/ASAN timeout and QPS rate adjustments #5321
Conversation
tsan documentation says 2-20x, so set it at 20x asan documentation says 1.2-2.7x, so set it at 3x msan documentation says 2-4x, so set it at 4x This is now much less optimistic than before 2. Reactive tsan tests for qps_test 3. Set CPU load for qps_openloop_test 4. Divide qps_openloop_test Poisson rate by the slowdown factor of the configuration
|
Can one of the admins verify this patch? |
|
Can one of the admins verify this patch? |
|
Thanks for looking at this. The worry here is test latency. Indeed the first run of this reports 6.5 hours for the TSAN run. I don't believe that number however: I think something deeper is going on - individual tests don't seem that much slower. I just ran it again to get some unloaded latency numbers - since I'm not sure anyone else is likely to run tests at 4am. Let's figure out:
So far I've been playing the game of precariously balancing then via the timeout multiplier simply for expediency (and that it mostly worked). |
|
So it seems that total time taken is sum(individual_test_time), which is of course bogus (we run tests in parallel). |
|
LGTM. How are you feeling on this one? |
|
I think I feel ok with it. 5 seems to be faster than 20 but still not flaking. |
TSAN/ASAN timeout and QPS rate adjustments
The goal here is to reduce the likelihood of flakes caused by timeout errors
tsan documentation says 2-20x, so set it at 20x
asan documentation says 1.2-2.7x, so set it at 3x
msan documentation says 2-4x, so set it at 4x
This is now much less optimistic than before
the configuration