Udacity_control

Submission for completing the Udacity Project

Implementation

PPO

• GAE returns
• Network - shared torso, dual head.

There are two Environments:

Reacher:

The agent controls a robotic arm with a goal to have the point of the arm reside in the goal zone. Each tick that the arm resides in the goal zone, the agent gets a 0.1 reward. Otherwise the agent gets 0 reward.

There are two types of Reacher environments. One with a single agent, and one with 20 agents. The observations are unique to each agent. The action space is continuous.

• State space = Array of size (33)
• Action space (Continuous) = Array of size (4), each action between -1,1

---

Optional environment

Crawler:

• Set-up: A creature with 4 arms and 4 forearms.
• Goal: The agents must move its body toward the goal direction without falling.
  • CrawlerStaticTarget - Goal direction is always forward.
  • CrawlerDynamicTarget- Goal direction is randomized.
• Agents: The environment contains 3 agent linked to a single Brain.
• Agent Reward Function (independent):
• +0.03 times body velocity in the goal direction.
• +0.01 times body direction alignment with goal direction.
• Brains: One Brain with the following observation/action space.

• Vector Observation space: 117 variables corresponding to position, rotation, velocity, and angular velocities of each limb plus the acceleration and angular acceleration of the body.
  

• Vector Action space: (Continuous) Size of 20, corresponding to target rotations for joints.
  

• Visual Observations: None.
  

• Reset Parameters: None

The environment is considered solved under the following conditions:

• Benchmark Mean Reward for CrawlerStaticTarget: 2000
• Benchmark Mean Reward for CrawlerDynamicTarget: 400

Installation

Clone the repository.

git clone git@github.com:MorGriffiths/Udacity_Navigation.git
cd Udacity_Navigation

Install Unity ml-agents.

git clone https://github.com/Unity-Technologies/ml-agents.git
git -C ml-agents checkout 0.4.0b
pip install ml-agents/python/.

install anaconda

install the anaconda environment from the conda_requirements.txt file

conda create --name Reacher --file conda_requirements.txt

depending on which version of anaconda you have

conda activate Reacher

or

source activate Reacher

Download the Reacher Unity Environment which matches your operating system

• Linux
• Mac OSX
• Windows (32-bits)
• Windows (64 bits)

Download the Crawler Unity Environment

• Linux
• Mac OSX
• Windows (32-bits)
• Windows (64 bits)

Place the environment into the Environments folder. If necessary, inside main.py, change the path to the unity environment appropriately

Project Layout

Agents

PPO, The rest are either unfinished or are in need of learning help (D4PG,DDPG,REINFORCE)

Buffers

ReplayBuffer, PriorityReplayBuffer (Uses Priority tree)

Networks

Policy (PPO), Actor/Critic (DDPG), ReinforcePolicy (REINFORCE)

Agent weights

model_checkpoints/ppo_checkpoint.pth

Run the project

Make sure the env path is set correctly in the main.py file and run

python main.py

Every 10 episodes the performance plot is updated "PPO_performance.png" so you can watch how the agent performs along with the readouts from the terminal

Watch a trained agent

On Ubuntu this results in a laughably tiny window. Haven't dug into why this isn't working very well.

python watch_agent.py