Feature/ Smac wrapper Update, MADQN/QMIX/VDN upgrades and Dockerfile improvements #310

KaleabTessera · 2021-09-01T11:56:51Z

What?

Smac & Wrapper:
- Switched to PZ SMAC wrapper (https://github.com/oxwhirl/smac/tree/master/smac/examples/pettingzoo).
- Minor bug fixes to PZ core wrapper.
MADQN/QMIX/VDN:
- Updates and minor fixes to action selectors.
- Moved epsilon decay out of trainers to executors and evaluators.
- Added support for timestep based epsilon decay e.g. decay for 50000 time steps.
All Systems:
- Added optional LR decay and examples of usage.
Other:
- Redesigned dockerfiles (i.e. dockerfile per env) & makefiles. @DriesSmit @arnupretorius @mmorris44 Please check the redesigned dockerfiles :)
- Tested and updated versions of most packages.
- Changed how github actions are done.

Why?

Fixes and more features.

How?

Extra

…format action masks.

…pgrades

…feature/smac-env-upgrades

…v-upgrades

…pgrades

mmorris44 · 2021-11-24T14:09:42Z

Remaining problems:

Robocup sudo not found -> fix by installing sudo in Dockerfile: RUN apt-get -y install sudo -> install now works, but run broken, can't find server file -> Robocup example itself is broken (Dries fixed in another PR), but Robocup build also broken
SC2 working fine -> turned out to just be a couple of faulty downloads on my side

mmorris44 · 2021-11-25T10:04:50Z

Error during robocup install that gets ignored, probably causing the issues:

#15 264.1 libtool: link: g++ -std=c++14 -W -Wall -g -O2 -o .libs/rcssserver audio.o bodysender.o coach.o csvsaver.o
 dispsender.o field.o fullstatesender.o heteroplayer.o initsender.o initsendercoach.o initsenderlogger.o initsender
monitor.o initsenderonlinecoach.o initsenderplayer.o landmarkreader.o logger.o main.o monitor.o pcombuilder.o pcomp
arser.o player.o playerparam.o object.o referee.o remoteclient.o resultsaver.o serializer.o serializercoachstdv1.o 
serializercoachstdv7.o serializercoachstdv8.o serializercoachstdv13.o serializercoachstdv14.o serializercommonstdv1
.o serializercommonstdv7.o serializercommonstdv8.o serializermonitor.o serializeronlinecoachstdv1.o serializeronlin
ecoachstdv6.o serializeronlinecoachstdv7.o serializeronlinecoachstdv8.o serializeronlinecoachstdv13.o serializeronl
inecoachstdv14.o serializerplayerstdv1.o serializerplayerstdv7.o serializerplayerstdv8.o serializerplayerstdv13.o s
erializerplayerstdv14.o serverparam.o stadium.o stdoutsaver.o stdtimer.o synctimer.o team.o utility.o visualsenderc
oach.o visualsenderplayer.o weather.o xmlreader.o xpmholder.o player_command_parser.o player_command_tok.o  -L../rc
ssbase/conf -L../rcssbase/net -L../rcssbase/gzip -L/usr/lib/x86_64-linux-gnu -lrcssclangparser /home/app/mava/rcsss
erver-rcssserver-16.0.0/rcssbase/conf/.libs/librcssconfparser.so /home/app/mava/rcssserver-rcssserver-16.0.0/rcssba
se/net/.libs/librcssnet.so /home/app/mava/rcssserver-rcssserver-16.0.0/rcssbase/gzip/.libs/librcssgz.so -lboost_fil
esystem -lboost_system -lz -lm
#15 264.3 /usr/bin/ld: cannot find -lrcssclangparser
#15 264.3 collect2: error: ld returned 1 exit status
#15 264.3 make[3]: *** [Makefile:913: rcssserver] Error 1
#15 264.3 make[3]: Leaving directory '/home/app/mava/rcssserver-rcssserver-16.0.0/src'
#15 264.3 make[2]: *** [Makefile:778: all] Error 2
#15 264.3 make[2]: Leaving directory '/home/app/mava/rcssserver-rcssserver-16.0.0/src'
#15 264.3 make[1]: *** [Makefile:439: all-recursive] Error 1
#15 264.3 make[1]: Leaving directory '/home/app/mava/rcssserver-rcssserver-16.0.0'
#15 264.3 make: *** [Makefile:371: all] Error 2
#15 264.3 --2021-11-25 09:58:11--  https://github.com/rcsoccersim/rcssmonitor/archive/rcssmonitor-16.0.0.tar.gz
#15 264.3 Resolving github.com (github.com)... 140.82.121.4
#15 264.3 Connecting to github.com (github.com)|140.82.121.4|:443... connected.```

…grades

mmorris44 · 2021-12-01T07:27:09Z

bash_scripts/install_robocup.sh

@@ -1,4 +1,4 @@
-sudo apt-get update \
+apt-get update \


I assume the reason for this change is that the docker container doesn't need root access to run these commands. However, won't making this change cause problems for when people try to install robocup outside of docker?

Good point. I will re-add sudo to the install. 👍

…pgrades

…va into feature/smac-env-upgrades

mmorris44

Can't find any further things that I would change, but my Mava knowledge isn't good enough for this to be terribly meaningful.
Nevertheless, great job @KaleabTessera!

DriesSmit

This PR looks good, thanks @KaleabTessera 🔥 A lot of changes in here. Did you run some regression tests on this PR?

arnupretorius

This is a big one @KaleabTessera! Looks good to me, a massive effort 🔥 Good that we had multiple iterations across sections of the code because it is a big PR. @mmorris44 also thanks for your detailed checks and testing some of the changes. In the future, we can perhaps try to keep the PRs smaller and just more frequent.

For reference - Kale-ab confirmed we have test runs for the new smac wrapper and the eps scheduling changes for MADQN/QMIX/VDN and the learning rate decay which all look good.

KaleabTessera added 25 commits August 4, 2021 15:28

feat: Log smac metrics such as win_rate and use existing renderer.

97074fe

feat: Updated smac wrapper to better log agent details and correctly …

8e5256e

…format action masks.

Merge remote-tracking branch 'origin/develop' into feature/smac-env-u…

96161bf

…pgrades

wip(smac): WIP new smac wrapper.

3c09aa1

Merge remote-tracking branch 'origin/develop' into feature/smac-env-u…

21a17c7

…pgrades

fix: Updated eps greedy module and add state info to PZ wrapper.

c746820

fix(wrappers): Updates to PZ wrappers to handle dying agents.

41221a9

Merge remote-tracking branch 'origin/develop' into feature/smac-env-u…

af664dc

…pgrades

chore: Consistency in loss name.

a65f013

feat(madqn,vdn): Upgraded action selectors in madqn and vdn.

db12989

Merge remote-tracking branch 'origin/feature/smac-env-upgrades' into …

0de351e

…feature/smac-env-upgrades

feat(qmix): Upgraded action selectors in qmix.

99ec5e1

feat(dial): Upgraded action selectors in dial.

79ed791

feat(examples): Updated custom madqn example.

b13308f

feat(qmix,vdn): Fixes to creation of agent networks.

e61329d

Merge branch 'chore/update-for-new-acme-version' into feature/smac-en…

132f3c4

…v-upgrades

feat(qmix,vdn,qmix): Added epsilon scheduling by time_t.

c79fff8

chore: Updated mypy.

562e97a

chore: Changes according to mypy typing.

ffae883

feat: Added lr decay to all systems.

a458d38

feat: Improved lr decay scheduling.

946b903

chore: Improved docstrings and lr decay examples.

4f10642

chore: Cleanup docstrings and update tests.

fbfe687

fix: Pass learning_rate_schedule in qmix.

983315d

Merge remote-tracking branch 'origin/develop' into feature/smac-env-u…

ba61dc1

…pgrades

KaleabTessera requested review from arnupretorius and DriesSmit as code owners September 1, 2021 11:56

KaleabTessera added 2 commits September 1, 2021 14:54

fix: Handle tuple observations.

a913b13

fix: Update spec from box to compatible spec.

59d1575

KaleabTessera self-assigned this Sep 1, 2021

Upgrade flatland to version 3

a1e215b

KaleabTessera added 12 commits November 29, 2021 13:38

docs: Updated readme for examples.

ca58de9

chore: Fixed formatting.

1053340

merge: Merge branch 'feature/eval-intervals' into feature/smac-env-up…

335353c

…grades

Merge branch 'feature/eval-intervals' into feature/smac-env-upgrades

6d4d043

chore: Updated flatland dependency.

bf940f4

feat: Updated dockerfile.

9c7a9e6

fix: Base image alias.

39543c1

fix: Set DEBIAN_FRONTEND to noninteractive to fix installs.

3df8642

fix: Updated robocup install script.

0ccdd05

fix: Updates to robocup dockerfile for ubuntu 20.04.

b543fda

feat: Added workflows for main/master branch.

2feb3db

fix: Updated robocup install script.

a1cc4ab

mmorris44 reviewed Dec 1, 2021

View reviewed changes

KaleabTessera added 9 commits December 1, 2021 09:48

fix: Re-add sudo to install robocup incase it is run outside of docker.

ef7d6e5

fix: Added sudo to robocup dockerfile.

c2fe2f0

feat: Upgraded version of rcssserver.

6739ac8

Merge remote-tracking branch 'origin/develop' into feature/smac-env-u…

9c829dc

…pgrades

chore: Removed deprecated dockerfile.

7e5273b

feat: Simplified meltingpot install.

50e3054

Merge branch 'feature/smac-env-upgrades' of github.com:instadeepai/Ma…

d15ea02

…va into feature/smac-env-upgrades

docs: Updated meltingpot readme.

0afc111

fix: Minor updates to meltingpot install.

27f55f7

mmorris44 approved these changes Dec 2, 2021

View reviewed changes

DriesSmit approved these changes Dec 2, 2021

View reviewed changes

arnupretorius approved these changes Dec 2, 2021

View reviewed changes

arnupretorius merged commit e4f9532 into develop Dec 2, 2021

arnupretorius deleted the feature/smac-env-upgrades branch December 2, 2021 09:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/ Smac wrapper Update, MADQN/QMIX/VDN upgrades and Dockerfile improvements #310

Feature/ Smac wrapper Update, MADQN/QMIX/VDN upgrades and Dockerfile improvements #310

KaleabTessera commented Sep 1, 2021 •

edited

Loading

mmorris44 commented Nov 24, 2021 •

edited

Loading

mmorris44 commented Nov 25, 2021 •

edited

Loading

mmorris44 Dec 1, 2021

KaleabTessera Dec 1, 2021

mmorris44 left a comment

DriesSmit left a comment

arnupretorius left a comment

Feature/ Smac wrapper Update, MADQN/QMIX/VDN upgrades and Dockerfile improvements #310

Feature/ Smac wrapper Update, MADQN/QMIX/VDN upgrades and Dockerfile improvements #310

Conversation

KaleabTessera commented Sep 1, 2021 • edited Loading

What?

Why?

How?

Extra

mmorris44 commented Nov 24, 2021 • edited Loading

mmorris44 commented Nov 25, 2021 • edited Loading

mmorris44 Dec 1, 2021

Choose a reason for hiding this comment

KaleabTessera Dec 1, 2021

Choose a reason for hiding this comment

mmorris44 left a comment

Choose a reason for hiding this comment

DriesSmit left a comment

Choose a reason for hiding this comment

arnupretorius left a comment

Choose a reason for hiding this comment

KaleabTessera commented Sep 1, 2021 •

edited

Loading

mmorris44 commented Nov 24, 2021 •

edited

Loading

mmorris44 commented Nov 25, 2021 •

edited

Loading