Could you please explain in detail how you get Max Average Return in Fig.5 & Table 1. in your paper.
From what I understand, if I want you evaluate one algorithm(like TD3) in one game and get statistics:

test_reward[10][1M/5k = 200]
for seed in range(10):
    set the seed
    for every 5k steps collected:
        test the policy for 10 times, record average game return in test_reward[seed][epoch]

now I have 10*200 = 2000 rewards in total, each reward represents average return over 10 test trails.
How exactly do you calculate Max Average Return and standard deviation.
For example, is it
max average return = (max(mean(test_reward, axis = 0)))
or
max average return = (mean(max(test_reward, axis = 0)))

same for deviation, thanks a lot