Overview

A2C agents interacting with the environment.

D4PG agents interacting with the environment.

This report discusses two algorithms, Advantage Actor-Critic (A2C) and Distributed Distributional Deep Deterministic Policy Gradients (D4PG), in the context of a continuous control reinforcement learning task. The agent is a double-jointed arm that is rewarded for keeping its hand inside a moving target location. The environment specifications are:

• 33 continuous-valued states corresponding to the position, rotation, velocity, and angular velocity of the arm.
• 4 continuous-valued actions corresponding to the torques applied to the two joints that fall in the range [-1, 1].
• A reward of 0.04 for each time step that the agent has its hand in the target location, and a reward of 0 otherwise.

The agent’s goal is to achieve an average score of 30 over a window of 100 episodes.

The code for the two algorithms can be found here: https://github.com/tyranitar/continuous-control.

Results

Average score across 20 parallel agents for A2C.

Average score across 20 parallel agents for D4PG.

For both A2C and D4PG, 20 parallel agents were used to collect trajectories. The goal was to achieve an average score of 30 over a window of 100 episodes across all 20 agents.

Given this goal, the A2C algorithm was able to solve the environment in 144 episodes. And while the learning was not as stable, D4PG was able to solve the environment faster, in 112 episodes.

This report breaks down both the A2C and D4PG algorithms and compares the two algorithms in terms of convergence behavior and learning speed.

Implementation

Architectures

2. ReLU
3. Linear 128 → 64
4. ReLU
5. Linear 64 → 4
6. Tanh
7. Sample with standard dev | 1. Linear 33 → 400
8. Batch norm
9. ReLU
10. Linear 400 → 300
11. Batch norm
12. ReLU
13. Linear 300 → 4
14. Tanh | | Critic | 1. Linear 33 → 128
15. ReLU
16. Linear 128 → 64
17. ReLU
18. Linear 64 → 1 | 1. Linear 33 → 400
19. Batch norm
20. ReLU
21. Concatenate actions
22. Linear 404 → 300
23. Batch norm
24. ReLU
25. Linear 300 → 51
26. Softmax |

Hyperparameters

For both the A2C actor and critic, the RMSProp optimizer was used. For both the D4PG actor and critic, the Adam optimizer was used.

A2C algorithm