Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to set the "nb_steps_warmup" and "nb_steps" properly? #386

Open
Beliefuture opened this issue Apr 4, 2022 · 1 comment
Open

How to set the "nb_steps_warmup" and "nb_steps" properly? #386

Beliefuture opened this issue Apr 4, 2022 · 1 comment

Comments

@Beliefuture
Copy link

I got the following warning while using keras-rl to train a DQN agent.
The batch_size argument is a small value i.e. 50.

According to the warning information below, it seems that there are not enough experience to sample.
I am not so sure about how to balance the arguments nb_steps_warmup and nb_steps.

Moreover, does the nb_steps_warmup is a part of nb_steps, i.e. if nb_steps_warmup=10, nb_steps=50, there remain 50-10=40 steps used to train?

/lib/python3.7/site-packages/rl/memory.py:37: UserWarning: Not enough entries to sample
 without replacement. Consider increasing your warm-up phase to avoid oversampling!
  warnings.warn('Not enough entries to sample without replacement. Consider increasing your warm-up phase to avoid overs
ampling!')
Traceback (most recent call last):
  File "main.py", line 124, in <module>
    history = agent.fit(env, nb_steps=args.nb_steps_train, visualize=False, verbose=args.verbose)
  File "/lib/python3.7/site-packages/rl/core.py", line 193, in fit
    metrics = self.backward(reward, terminal=done)
  File "/lib/python3.7/site-packages/rl/agents/dqn.py", line 250, in backward
    experiences = self.memory.sample(self.batch_size)
  File "/lib/python3.7/site-packages/rl/memory.py", line 263, in sample
    batch_idxs = sample_batch_indexes(0, self.nb_entries, size=batch_size)
  File "/lib/python3.7/site-packages/rl/memory.py", line 38, in sample_batch_indexes
    batch_idxs = np.random.random_integers(low, high - 1, size=size)
  File "mtrand.pyx", line 1328, in numpy.random.mtrand.RandomState.random_integers
  File "mtrand.pyx", line 746, in numpy.random.mtrand.RandomState.randint
  File "_bounded_integers.pyx", line 1254, in numpy.random._bounded_integers._rand_int64
ValueError: low >= high
@Beliefuture
Copy link
Author

Beliefuture commented Apr 4, 2022 •
edited
Loading

Another question is the meaning of train_interval on this issue #109 but no answer yet. The official documentation is not so detailed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Beliefuture