YAAF: Yet Another Agents Framework

Autonomous, Agents, Reinforcement, Learning, Deep, autonomous-agents, deep-learning, reinforcement-learning
pip install yaaf==1.1.0


Yet Another Agents Framework

An RL research-oriented framework for agent prototyping and evaluation


YAAF is a reinforcement learning, research-oriented framework designed for quick agent prototyping and evaluation.

At its core, YAAF follows the assumptions that:

  • Agents execute actions upon the environment (which yields observations and rewards in return)
  • Environments follow the interface from OpenAI Gym
  • Agents follow a clear agent interface
  • Any deep learning framework can be used for deep rl agents (even though it comes packed with PyTorch tools)

As a simple example, suppose you'd want to evaluate an agent following a random policy on the Space Invaders environment

import gym

from yaaf.agents import RandomAgent
from yaaf.evaluation import AverageEpisodeReturnMetric
from yaaf.execution import EpisodeRunner
from yaaf.visualization import LinePlot

env = gym.make("SpaceInvaders-v0")
agent = RandomAgent(num_actions=env.action_space.n)
metric = AverageEpisodeReturnMetric()
runner = EpisodeRunner(5, agent, env, [metric], render=True).run()

plot = LinePlot("Space Invaders Random Policy", x_label="Episode", y_label="Average Episode Return", num_measurements=5)
plot.add_run("random policy", metric.result())

Quick Disclaimer:

YAAF is not yet another deep reinforcement learning framework.

If you are looking for high-quality implementations of state-of-the-art algorithms, then I suggest the following libraries:


For the first installation I suggest setting up new Python 3.7 virtual environment

$ python -m venv yaaf_test_environment
$ source yaaf_test_environment/bin/activate
$ pip install --upgrade pip setuptools
$ pip install yaaf  
$ pip install gym[atari] # Optional - Atari2600


1 - Space Invaders DQN
import gym
from yaaf.environments.wrappers import DeepMindAtari2600Wrapper
from yaaf.agents.dqn import DeepMindAtariDQNAgent
from yaaf.execution import TimestepRunner
from yaaf.evaluation import AverageEpisodeReturnMetric, TotalTimestepsMetric

env = DeepMindAtari2600Wrapper(gym.make("SpaceInvaders-v0"))
agent = DeepMindAtariDQNAgent(num_actions=env.action_space.n)

metrics = [AverageEpisodeReturnMetric(), TotalTimestepsMetric()]
runner = TimestepRunner(1e9, agent, env, metrics, render=True).run()
2 - CartPole DQN
import gym
from yaaf.agents.dqn import MLPDQNAgent
from yaaf.execution import EpisodeRunner
from yaaf.evaluation import AverageEpisodeReturnMetric, TotalTimestepsMetric

env = gym.make("CartPole-v0")
layers = [(64, "relu"), (64, "relu")]
agent = MLPDQNAgent(num_features=env.observation_space.shape[0], num_actions=env.action_space.n, layers=layers)

metrics = [AverageEpisodeReturnMetric(), TotalTimestepsMetric()]
runner = EpisodeRunner(100, agent, env, metrics, render=True).run()
3 - Asynchronous Advantage Actor-Critic on GPU
(my multi-task implementation, requires tensorflow-gpu)


from yaaf.environments.wrappers import NvidiaAtari2600Wrapper
from yaaf.agents.hga3c import HybridGA3CAgent
from yaaf.execution import AsynchronousParallelRunner

num_processes = 8

envs = [NvidiaAtari2600Wrapper("SpaceInvadersDeterministic-v4")
        for _ in range(num_processes)

hga3c = HybridGA3CAgent(
            environment_names=[env.spec.id for env in envs],
            environment_actions=[env.action_space.n for env in envs],

trainer = AsynchronousParallelRunner(
    render_ids=[0, 1, 2]

while trainer.running:

4 - CartPole DQN from scratch
import gym
from yaaf.agents.dqn import DQNAgent
from yaaf.agents.dqn.networks import DeepQNetwork

from yaaf.execution import EpisodeRunner
from yaaf.evaluation import AverageEpisodeReturnMetric, TotalTimestepsMetric
from yaaf.models.feature_extraction import MLPFeatureExtractor

# Setup env
env = gym.make("CartPole-v0")
num_features = env.observation_space.shape[0]
num_actions = env.action_space.n

# Setup model
mlp_feature_extractor = MLPFeatureExtractor(num_inputs=num_features, layers=[(64, "relu"), (64, "relu")])
network = DeepQNetwork(feature_extractors=[mlp_feature_extractor],
                       num_actions=num_actions, learning_rate=0.001, optimizer="adam", cuda=True)
# Setup agent
agent = DQNAgent(network, num_actions, 
                 discount_factor=0.95, initial_exploration_steps=1000, final_exploration_rate=0.001)

# Run
metrics = [AverageEpisodeReturnMetric(), TotalTimestepsMetric()]
runner = EpisodeRunner(100, agent, env, metrics, render=True).run()

Markov Sub-module


Citing the Project

When using YAAF in your projects, cite using:

  author = {João Ribeiro},
  title = {YAAF - Yet Another Agents Framework},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/jmribeiro/yaaf}},


  • Documentation
  • Code cleanup
  • More algorithms


If you want to contribute to this project, feel free to contact me by e-mail or open an issue.


YAAF was developed as a side-project to my research work and its creation was motivated by work done in the project Ad Hoc Teams With Humans And Robots funded by the Air Force Office of Scientific Research, in collaboration with PUC-Rio.