FIDLE_rl_baselines_zoo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XJy9QoDC7XA7"
      },
      "source": [
        "<img width=\"800px\" src=\"../fidle/img/header.svg\"></img>\n",
        "\n",
        "# <!-- TITLE --> [DRL2] - RL Baselines3 Zoo: Training in Colab\n",
        "<!-- DESC --> Demo of Stable baseline3 with Colab\n",
        "<!-- AUTHOR : Nathan Cassereau (IDRIS) and Bertrand Cabot (IDRIS) -->\n",
        "\n",
        "\n",
        "Demo of Stable baseline3 adapted By Nathan Cassereau (IDRIS) and Bertrand Cabot (IDRIS)\n",
        "\n",
        "\n",
        "Github Repo: [https://github.com/DLR-RM/rl-baselines3-zoo](https://github.com/DLR-RM/rl-baselines3-zoo)\n",
        "\n",
        "Stable-Baselines3 Repo: [https://github.com/DLR-RM/rl-baselines3-zoo](https://github.com/DLR-RM/stable-baselines3)\n",
        "\n",
        "\n",
        "# Install Dependencies\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AXVDDlTn02M9"
      },
      "outputs": [],
      "source": [
        "!apt-get install swig cmake ffmpeg freeglut3-dev xvfb"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f1s9_3b1Tsq9"
      },
      "outputs": [],
      "source": [
        "!apt-get install -y \\\n",
        "    libgl1-mesa-dev \\\n",
        "    libgl1-mesa-glx \\\n",
        "    libglew-dev \\\n",
        "    libosmesa6-dev \\\n",
        "    software-properties-common\n",
        "\n",
        "!apt-get install -y patchelf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kDjF3qRg7oGH"
      },
      "source": [
        "## Clone RL Baselines3 Zoo Repo"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SCjGikdT1DFy"
      },
      "outputs": [],
      "source": [
        "!git clone --recursive https://github.com/DLR-RM/rl-baselines3-zoo"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "REMQlh-ezyVt"
      },
      "outputs": [],
      "source": [
        "%cd /content/rl-baselines3-zoo/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5tmD_QTBqTMb"
      },
      "source": [
        "### Install pip dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OWIDzgJTqShY"
      },
      "outputs": [],
      "source": [
        "!pip install -r requirements.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VlOD-cImRMwW"
      },
      "outputs": [],
      "source": [
        "!pip install free-mujoco-py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kYPotFDF0Noa"
      },
      "source": [
        "## Pretrained model\n",
        "\n",
        "gym environments: https://gym.openai.com/envs/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kgOp2XIklaf2"
      },
      "outputs": [],
      "source": [
        "%cd /content/rl-baselines3-zoo/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xVm9QPNVwKXN"
      },
      "source": [
        "### Record  a Video"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MPyfQxD5z26J"
      },
      "outputs": [],
      "source": [
        "# Set up display; otherwise rendering will fail\n",
        "import os\n",
        "os.system(\"Xvfb :1 -screen 0 1024x768x24 &\")\n",
        "os.environ['DISPLAY'] = ':1'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZC3OTfpf8CXu"
      },
      "outputs": [],
      "source": [
        "import base64\n",
        "from pathlib import Path\n",
        "\n",
        "from IPython import display as ipythondisplay\n",
        "\n",
        "def show_videos(video_path='', prefix=''):\n",
        "  \"\"\"\n",
        "  Taken from https://github.com/eleurent/highway-env\n",
        "\n",
        "  :param video_path: (str) Path to the folder containing videos\n",
        "  :param prefix: (str) Filter the video, showing only the only starting with this prefix\n",
        "  \"\"\"\n",
        "  html = []\n",
        "  for mp4 in Path(video_path).glob(\"**/*{}*.mp4\".format(prefix)):\n",
        "      video_b64 = base64.b64encode(mp4.read_bytes())\n",
        "      html.append('''{} <br> <video alt=\"{}\" autoplay \n",
        "                    loop controls style=\"height: 400px;\">\n",
        "                    <source src=\"data:video/mp4;base64,{}\" type=\"video/mp4\" />\n",
        "                </video>'''.format(mp4, mp4, video_b64.decode('ascii')))\n",
        "  ipythondisplay.display(ipythondisplay.HTML(data=\"<br>\".join(html)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LW-7EWA50550"
      },
      "source": [
        "### Discrete environments"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ArN0B6YX0m4z"
      },
      "outputs": [],
      "source": [
        "%run scripts/all_plots.py -a dqn qrdqn a2c ppo --env PongNoFrameskip-v4 -f rl-trained-agents/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5zmVA6JADKvV"
      },
      "outputs": [],
      "source": [
        "%run scripts/plot_train.py -a dqn -e PongNoFrameskip-v4 -f rl-trained-agents/ -x time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BkcI91hABKUm"
      },
      "outputs": [],
      "source": [
        "%run scripts/plot_train.py -a qrdqn -e PongNoFrameskip-v4 -f rl-trained-agents/ -x time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "E5s7UHn2DmqQ"
      },
      "outputs": [],
      "source": [
        "%run scripts/plot_train.py -a a2c -e PongNoFrameskip-v4 -f rl-trained-agents/ -x time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0aP2A-NqCQSC"
      },
      "outputs": [],
      "source": [
        "%run scripts/plot_train.py -a ppo -e PongNoFrameskip-v4 -f rl-trained-agents/ -x time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "itP336W6HdeC"
      },
      "outputs": [],
      "source": [
        "!python enjoy.py --algo dqn --env PongNoFrameskip-v4 --no-render --n-timesteps 5000"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ip3AauLzwNGP"
      },
      "outputs": [],
      "source": [
        "!python -m utils.record_video --algo dqn --env PongNoFrameskip-v4"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oKOjFuwK9HI0"
      },
      "outputs": [],
      "source": [
        "show_videos(video_path='rl-trained-agents/dqn', prefix='PongNoFrameskip-v4')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8kD_C440-xvw"
      },
      "source": [
        "### Continuous environments"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2yJ0eOyjLM5O"
      },
      "outputs": [],
      "source": [
        "%run scripts/all_plots.py -a ppo trpo sac td3 tqc --env Ant-v3 -f rl-trained-agents/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iQAJmndtNCvN"
      },
      "outputs": [],
      "source": [
        "%run scripts/plot_train.py -a ppo -e Ant-v3 -f rl-trained-agents/ -x time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O1L_ze67NDf9"
      },
      "outputs": [],
      "source": [
        "%run scripts/plot_train.py -a trpo -e Ant-v3 -f rl-trained-agents/ -x time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FPn0iySFNzCQ"
      },
      "outputs": [],
      "source": [
        "%run scripts/plot_train.py -a tqc -e Ant-v3 -f rl-trained-agents/ -x time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3vDHeNNTNwsT"
      },
      "outputs": [],
      "source": [
        "%run scripts/plot_train.py -a td3 -e Ant-v3 -f rl-trained-agents/ -x time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dQDNhdlgOzFs"
      },
      "outputs": [],
      "source": [
        "%run scripts/plot_train.py -a sac -e Ant-v3 -f rl-trained-agents/ -x time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s9aT46m8QXzt"
      },
      "outputs": [],
      "source": [
        "!python enjoy.py --algo td3 --env Ant-v3 --no-render --n-timesteps 5000"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zx7gZ23UQYQ4"
      },
      "outputs": [],
      "source": [
        "!python -m utils.record_video --algo td3 --env Ant-v3"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yvETH_M_Unw3"
      },
      "outputs": [],
      "source": [
        "show_videos(video_path='rl-trained-agents/td3', prefix='Ant-v3')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6gJ-pAbF7zRZ"
      },
      "source": [
        "## Train an RL Agent\n",
        "\n",
        "\n",
        "The train agent can be found in the `logs/` folder.\n",
        "\n",
        "Here we will train A2C on CartPole-v1 environment for 100 000 steps. \n",
        "\n",
        "\n",
        "To train it on Pong (Atari), you just have to pass `--env PongNoFrameskip-v4`\n",
        "\n",
        "Note: You need to update `hyperparams/algo.yml` to support new environments. You can access it in the side panel of Google Colab. (see https://stackoverflow.com/questions/46986398/import-data-into-google-colaboratory)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "9bIR_N7R11XI"
      },
      "outputs": [],
      "source": [
        "!python train.py --algo dqn --env PongNoFrameskip-v4 --n-timesteps 1000000"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-fHBq73665yD"
      },
      "source": [
        "#### Evaluate trained agent\n",
        "\n",
        "\n",
        "You can remove the `--folder logs/` to evaluate pretrained agent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bw8YuEgU6bT3"
      },
      "outputs": [],
      "source": [
        "!python enjoy.py --algo dqn --env PongNoFrameskip-v4 --no-render --n-timesteps 5000 --folder logs/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w5Il2J0VHPLC"
      },
      "source": [
        "#### Tune Hyperparameters\n",
        "\n",
        "We use [Optuna](https://optuna.org/) for optimizing the hyperparameters.\n",
        "\n",
        "Tune the hyperparameters for PPO, using a tpe sampler and median pruner, 2 parallels jobs,\n",
        "with a budget of 1000 trials and a maximum of 50000 steps"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w2sC22eGHTH-"
      },
      "outputs": [],
      "source": [
        "#!python train.py --algo dqn --env PongNoFrameskip-v4 -n 5000 -optimize --n-trials 10 --n-jobs 5 --sampler tpe --pruner median"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qBuUfnzI8DN6"
      },
      "source": [
        "### Display the video"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RjdpP0HE8D2p"
      },
      "source": [
        "### Continue Training\n",
        "\n",
        "Here, we will continue training of the previous model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zgMZQJJF6u1C"
      },
      "outputs": [],
      "source": [
        "#!python train.py --algo dqn --env PongNoFrameskip-v4  --n-timesteps 50000 -i logs/dqn/PongNoFrameskip-v4_1/PongNoFrameskip-v4.zip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GSaoyiAE8cVj"
      },
      "outputs": [],
      "source": [
        "#!python enjoy.py --algo dqn --env PongNoFrameskip-v4 --no-render --n-timesteps 1000 --folder logs/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SKglp1awG6c6"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "FIDLE rl-baselines-zoo.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}