Feature

• HAPPO seems to be missing in the current implementation and either being WIP or discarded, even though HASAC was implemented in Mava already. So implementing HAPPO + HAPPO_rec would be a relevant feature.

• Heterogeneous agent versions of any I-type and MA-type Agent would be interesting for having heterogeneous agent counterparts of agents like IPPO and MAPPO, that could tackle tasks that require heterogeneity to solve them properly with the standard algorithms of IPPO/MAPPO.

  • For context Agents like HetIPPO were introduced in the VMAS paper, while a "HetMAPPO" version was used in the MAPPO Paper to allow the use of heterogeneous agents. I spotted TorchRL adopting these heterogeneous agent versions already (see this line), while they still miss agents like HAPPO or HASAC.
  • Heterogeneous agent versions would also enable better comparability between agents like (Het-)MAPPO and the more advanced HAPPO agent, which might be relevant for future research on the MARL agents, and in application.
  • HetMAPPO/HetIPPO means only not to share parameters in MAPPO/IPPO. So all that needs to be done to implement Het- agent versions, is to disable parameter sharing. HAPPO differs from HetMAPPO in its additional sequential updating scheme, while HetMAPPO simply updates all agents at once like MAPPO.

Proposal

• It seems that to implement HAPPO not much has to be changed from MAPPO apart from adding the sequential HARL updating scheme that (of my knowledge) only needs to calculate updating weights M as in the depiction below. Else, the implementation should be the same as MAPPO.
  

(Image Source)

I also looked through the Mava code and can't spot any barrier implementing this yet. One should essentially be able to implement this into the actor_loss_fn similarly to HASAC.

This documentation source might be helpful as additional context for HAPPO.

To implement Het- Agent versions, simply provide an option/implementation without parameter sharing for I- and MA-Type MARL agents. The recurrent versions for the Het-type agents might be kept up as well.

Definition of done

Implemented HAPPO, HAPPO_rec, HetIPPO, HetIPPO_rec, HetMAPPO_rec, HetISAC, HetMASAC.

One can do the same with IQL and QMIX, giving HetQMix_rec, HetIQL_rec - I wonder why there are no non-recurrent IQL/QMIX variants; I guess they do not perform that well as recurrent variants, but it could be still interesting to have a classic IQL/QMIX version for testing. [OPTIONAL]

In general, these are a lot of ideas/tasks. Feel free to just implement what seems relevant to you.