diff --git a/DRL/FIDLE_DQNfromScratch.ipynb b/DRL/FIDLE_DQNfromScratch.ipynb
new file mode 100755
index 0000000000000000000000000000000000000000..532d4c503294a97abd799ebefa0943febf06e0f9
--- /dev/null
+++ b/DRL/FIDLE_DQNfromScratch.ipynb
@@ -0,0 +1,541 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "w_5p3EyVknLC",
+ "metadata": {
+ "id": "w_5p3EyVknLC"
+ },
+ "source": [
+ "<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n",
+ "\n",
+ "# <!-- TITLE --> [DRL1] - Solving CartPole with DQN\n",
+ "<!-- DESC --> Using a a Deep Q-Network to play CartPole - an inverted pendulum problem (PyTorch)\n",
+ "<!-- AUTHOR : Nathan Cassereau (IDRIS) and Bertrand Cabot (IDRIS) -->\n",
+ "\n",
+ "\n",
+ "\n",
+ "By Nathan Cassereau (IDRIS) and Bertrand Cabot (IDRIS)\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ucB28wGpmFwi",
+ "metadata": {
+ "id": "ucB28wGpmFwi"
+ },
+ "source": [
+ "## Objectives\n",
+ "\n",
+ "* Understand the code behind the DQN algorithm\n",
+ "* Visualize the result for fun purposes :)\n",
+ "\n",
+ "This notebook implements a DQN from scratch and trains it. It is simply a vanilla DQN with a target network (sometimes referred as Double DQN). More sophisticated and recent modifications might help stabilize the training.\n",
+ "\n",
+ "Considering that we are going to use a tiny network for a simple environment, matrix multiplications are not that time consuming, and using a GPU can be detrimental as communications between CPU and GPU are no longer negligeable compared to forward and backward steps. This notebook will therefore be executed on CPU.\n",
+ "\n",
+ "The chosen environment will be imported from the gym toolkit (https://gym.openai.com/)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fqQsB2Jwm-BP",
+ "metadata": {
+ "id": "fqQsB2Jwm-BP"
+ },
+ "source": [
+ "## Demonstration steps:\n",
+ "\n",
+ "- Define numerous hyperparameters\n",
+ "- Implement the Q-Network\n",
+ "- Implement an agent following the Double DQN algorithm\n",
+ "- Train it for a few minutes\n",
+ "- Visualize the result"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "nRJmgZ0inpkk",
+ "metadata": {
+ "id": "nRJmgZ0inpkk"
+ },
+ "source": [
+ "## Installations\n",
+ "\n",
+ "Gym requires a graphical interface to render a state observation. Xvfb allows to run the notebook headless. This software is not available on Jean Zay's compute node, hence the usage of Google colab."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "y2Y71JbfgkeU",
+ "metadata": {
+ "id": "y2Y71JbfgkeU"
+ },
+ "outputs": [],
+ "source": [
+ "!pip3 install pyvirtualdisplay\n",
+ "!pip install pyglet==1.5.11\n",
+ "!apt-get install x11-utils > /dev/null 2>&1 \n",
+ "!apt-get install -y xvfb python-opengl > /dev/null 2>&1"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "q6eYfBKnoOJQ",
+ "metadata": {
+ "id": "q6eYfBKnoOJQ"
+ },
+ "source": [
+ "## Imports\n",
+ "\n",
+ "I chose to use Pytorch to implement this DQN due to its straightforward API and personal preferences.\n",
+ "Gym implements the environment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0fc91d65-4756-4432-906c-7d315d981775",
+ "metadata": {
+ "id": "0fc91d65-4756-4432-906c-7d315d981775"
+ },
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "\n",
+ "import torch\n",
+ "import torch.nn as nn\n",
+ "\n",
+ "import gym\n",
+ "from gym import wrappers\n",
+ "\n",
+ "import random\n",
+ "from tqdm.notebook import tqdm\n",
+ "\n",
+ "import functools\n",
+ "import matplotlib.pyplot as plt\n",
+ "import os\n",
+ "import io\n",
+ "import base64\n",
+ "import glob\n",
+ "from IPython.display import display, HTML"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "Hao-RYcdowHn",
+ "metadata": {
+ "id": "Hao-RYcdowHn"
+ },
+ "source": [
+ "## Hyperparameters\n",
+ "\n",
+ "The size of the replay buffer does not matter much. In this case, it is big enough to hold every transitions we will have in our training. This choice does have a huge impact on memory though.\n",
+ "\n",
+ "Warm-up allows the network to gather some information before the training process begins.\n",
+ "\n",
+ "The target network will only be updated once every 10k steps in order to stabilize the training.\n",
+ "\n",
+ "The exploration rate is linearly decreasing, although an exponential curve is a sound and common choice as well.\n",
+ "\n",
+ "As mentioned above, only the CPU will be used, the GPU would be useful for bigger networks, and / or environments which have a torch tensor internal state.\n",
+ "\n",
+ "Considering this is a simple DQN implementation, its stability leaves a lot to be desired. In order not to rely on luck, a decent seed was chosen."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6fX1X6y6YHXF",
+ "metadata": {
+ "id": "6fX1X6y6YHXF"
+ },
+ "outputs": [],
+ "source": [
+ "learning_rate = 0.0001\n",
+ "buffer_size = 200000\n",
+ "warmup_steps = 10000\n",
+ "batch_size = 32\n",
+ "gamma = 0.99\n",
+ "train_freq = 4\n",
+ "target_update_interval = 10000\n",
+ "exploration_fraction = 0.1\n",
+ "exploration_initial_eps = 1.0\n",
+ "exploration_final_eps = 0.05\n",
+ "device = torch.device(\"cpu\") # torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
+ "\n",
+ "seed = 987654321\n",
+ "np.random.seed(seed)\n",
+ "torch.manual_seed(seed)\n",
+ "random.seed(seed)\n",
+ "if torch.cuda.is_available():\n",
+ " torch.cuda.manual_seed(seed)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "TofGB-s7qfSH",
+ "metadata": {
+ "id": "TofGB-s7qfSH"
+ },
+ "source": [
+ "## Q-Network and Agent implementation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4VhftO9PaE9g",
+ "metadata": {
+ "id": "4VhftO9PaE9g"
+ },
+ "outputs": [],
+ "source": [
+ "class DQN(nn.Module):\n",
+ "\n",
+ " def __init__(self):\n",
+ " super(DQN, self).__init__()\n",
+ " self.layer1 = nn.Linear(4, 64)\n",
+ " self.layer2 = nn.Linear(64, 64)\n",
+ " self.layer3 = nn.Linear(64, 2)\n",
+ " self.relu = nn.ReLU()\n",
+ "\n",
+ " def forward(self, x):\n",
+ " x = self.relu(self.layer1(x))\n",
+ " x = self.relu(self.layer2(x))\n",
+ " return self.layer3(x)\n",
+ "\n",
+ " def compute_target(self, x, rewards):\n",
+ " with torch.no_grad():\n",
+ " values = torch.zeros(x.shape[0], device=device)\n",
+ " values[rewards != 1] = torch.max(self.forward(x[rewards != 1]), dim=-1)[0]\n",
+ " values = rewards + gamma * values\n",
+ " return values\n",
+ "\n",
+ " def predict(self, x):\n",
+ " if len(x.shape) < 2:\n",
+ " x = x[None, :]\n",
+ " with torch.no_grad():\n",
+ " x = torch.argmax(self.forward(x), dim=-1)\n",
+ " if x.device.type == \"cuda\":\n",
+ " x = x.cpu()\n",
+ " return x\n",
+ "\n",
+ "class Agent:\n",
+ "\n",
+ " def __init__(self, env):\n",
+ " self.env = env\n",
+ " self.q_network = DQN().to(device)\n",
+ " self.target_network = DQN().to(device)\n",
+ " self.target_network.eval()\n",
+ " self.synchronize()\n",
+ " self.optimizer = torch.optim.Adam(self.q_network.parameters(), lr=learning_rate)\n",
+ " self.criterion = nn.MSELoss()\n",
+ " self.buffer = []\n",
+ " self.n_updates = 0\n",
+ " \n",
+ " def add_transition(self, state, action, reward, nextState):\n",
+ " self.buffer.append((state, action, reward, nextState))\n",
+ " if len(self.buffer) > buffer_size:\n",
+ " self.buffer.pop(random.randrange(len(self.buffer)))\n",
+ "\n",
+ " def sample(self):\n",
+ " transitions = random.sample(self.buffer, batch_size)\n",
+ " states, actions, rewards, nextStates = zip(*transitions)\n",
+ " states = torch.stack(states).to(device)\n",
+ " actions = torch.cat(actions).to(device)\n",
+ " rewards = torch.cat(rewards).to(device)\n",
+ " nextStates = torch.stack(nextStates).to(device)\n",
+ " return states, actions, rewards, nextStates\n",
+ " \n",
+ " def train_step(self, step):\n",
+ " if step % target_update_interval == 0:\n",
+ " self.synchronize()\n",
+ " if step < warmup_steps or step % train_freq != 0:\n",
+ " return 0.\n",
+ "\n",
+ " states, actions, rewards, nextStates = self.sample()\n",
+ " output = self.q_network(states)\n",
+ " output = torch.gather(output, 1, actions.unsqueeze(-1)).view(-1)\n",
+ " expectedOutput = self.target_network.compute_target(nextStates, rewards).view(-1)\n",
+ " self.optimizer.zero_grad()\n",
+ " loss = self.criterion(output, expectedOutput)\n",
+ " loss.backward()\n",
+ " torch.nn.utils.clip_grad_norm_(self.q_network.parameters(), 10)\n",
+ " self.optimizer.step()\n",
+ " self.n_updates += 1\n",
+ " return loss.item()\n",
+ "\n",
+ " def synchronize(self):\n",
+ " self.target_network.load_state_dict(self.q_network.state_dict())\n",
+ "\n",
+ " def play(self, state, exploration_rate=0.):\n",
+ " if random.random() > exploration_rate:\n",
+ " return self.q_network.predict(state.to(device))\n",
+ " else:\n",
+ " shape = (state.shape[0],) if len(state.shape) > 1 else (1,)\n",
+ " return torch.randint(0, 2, size=shape)\n",
+ "\n",
+ " @functools.lru_cache(maxsize=None)\n",
+ " def exploration_slope(self, total_steps):\n",
+ " return (exploration_initial_eps - exploration_final_eps) / (exploration_fraction * total_steps)\n",
+ "\n",
+ " def exploration(self, step, total_steps):\n",
+ " eps = exploration_initial_eps - step * self.exploration_slope(total_steps)\n",
+ " return max(eps, exploration_final_eps)\n",
+ "\n",
+ " def train(self, total_steps):\n",
+ " obs = torch.from_numpy(env.reset()).float()\n",
+ "\n",
+ " n_episodes = 0\n",
+ " length_current_episode = 0\n",
+ " lengths = []\n",
+ " avg_reward = 0\n",
+ " loss_backup = 0.\n",
+ " acc_loss = 0.\n",
+ " acc_loss_count = 0\n",
+ " self.rewards = []\n",
+ "\n",
+ " with tqdm(range(total_steps), desc=\"Training agent\", unit=\"steps\") as pbar:\n",
+ " for step in pbar:\n",
+ " eps = self.exploration(step, total_steps)\n",
+ "\n",
+ " action = self.play(obs, eps)\n",
+ " new_obs, _, done, info = env.step(action.item())\n",
+ " reward = torch.tensor([1.0 if not done else -1.0], dtype=torch.float32)\n",
+ " new_obs = torch.from_numpy(new_obs).float()\n",
+ "\n",
+ " self.add_transition(obs, action, reward, new_obs)\n",
+ " loss = self.train_step(step)\n",
+ " if loss != 0:\n",
+ " acc_loss += loss\n",
+ " acc_loss_count += 1\n",
+ "\n",
+ " if done:\n",
+ " obs = torch.from_numpy(env.reset()).float()\n",
+ " n_episodes += 1\n",
+ " lengths.append(length_current_episode)\n",
+ " self.rewards.append(length_current_episode)\n",
+ " length_current_episode = 0\n",
+ " if len(lengths) >= 25:\n",
+ " avg_reward = sum(lengths) / len(lengths)\n",
+ " if acc_loss_count != 0:\n",
+ " loss_backup = acc_loss / acc_loss_count\n",
+ " else:\n",
+ " loss_backup = \"??\"\n",
+ " acc_loss = 0.\n",
+ " acc_loss_count = 0\n",
+ " lengths = []\n",
+ " else:\n",
+ " obs = new_obs\n",
+ " length_current_episode += 1\n",
+ "\n",
+ " pbar.set_postfix({\n",
+ " \"episodes\": n_episodes,\n",
+ " \"avg_reward\": avg_reward,\n",
+ " \"loss\": loss_backup,\n",
+ " \"exploration_rate\": eps,\n",
+ " \"n_updates\": self.n_updates,\n",
+ " })"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "Kne9b7vCql3N",
+ "metadata": {
+ "id": "Kne9b7vCql3N"
+ },
+ "source": [
+ "## Defining the environment"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "BXw4RmGpFkZm",
+ "metadata": {
+ "id": "BXw4RmGpFkZm"
+ },
+ "outputs": [],
+ "source": [
+ "env = gym.make(\"CartPole-v1\")\n",
+ "env.seed(seed+2)\n",
+ "env.reset()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "i93WQNsbqo68",
+ "metadata": {
+ "id": "i93WQNsbqo68"
+ },
+ "source": [
+ "## Training our agent"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "rAm6v_0HiEge",
+ "metadata": {
+ "id": "rAm6v_0HiEge"
+ },
+ "outputs": [],
+ "source": [
+ "agent = Agent(env)\n",
+ "agent.train(120000)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "PPT-tl4Rqroj",
+ "metadata": {
+ "id": "PPT-tl4Rqroj"
+ },
+ "source": [
+ "## Episodes length\n",
+ "\n",
+ "A very noisy curve. It does reach satisfying levels though."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "IoCnHaZKgHqI",
+ "metadata": {
+ "id": "IoCnHaZKgHqI"
+ },
+ "outputs": [],
+ "source": [
+ "fig = plt.figure(figsize=(20, 12))\n",
+ "plt.plot(agent.rewards)\n",
+ "plt.xlabel(\"Episodes\")\n",
+ "plt.ylabel(\"Episode length\")\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0fuolKppq1Ak",
+ "metadata": {
+ "id": "0fuolKppq1Ak"
+ },
+ "source": [
+ "## Result visualisation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "GXT1q5ckh0dG",
+ "metadata": {
+ "id": "GXT1q5ckh0dG"
+ },
+ "outputs": [],
+ "source": [
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "710b8294-4f75-49b5-a54a-777439ce8799",
+ "metadata": {
+ "id": "710b8294-4f75-49b5-a54a-777439ce8799"
+ },
+ "outputs": [],
+ "source": [
+ "env = gym.make(\"CartPole-v1\")\n",
+ "env.seed(4)\n",
+ "env = wrappers.Monitor(env, \"./CartPole-v1/\", force=True)\n",
+ "\n",
+ "obs = env.reset()\n",
+ "i = 0\n",
+ "\n",
+ "while True:\n",
+ " action = agent.q_network.predict(torch.from_numpy(obs).float().to(device))\n",
+ " \n",
+ " obs, rewards, done, info = env.step(action.item())\n",
+ " env.render()\n",
+ " if done:\n",
+ " break\n",
+ " else:\n",
+ " i += 1\n",
+ "env.close()\n",
+ "print(f\"Survived {i} steps\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c7ad6655-02b7-436e-a7ae-93a7222b100e",
+ "metadata": {
+ "id": "c7ad6655-02b7-436e-a7ae-93a7222b100e"
+ },
+ "outputs": [],
+ "source": [
+ "def ipython_show_video(path):\n",
+ " \"\"\"Shamelessly stolen from https://stackoverflow.com/a/51183488/9977878\n",
+ " \"\"\"\n",
+ " if not os.path.isfile(path):\n",
+ " raise NameError(\"Cannot access: {}\".format(path))\n",
+ "\n",
+ " video = io.open(path, 'r+b').read()\n",
+ " encoded = base64.b64encode(video)\n",
+ "\n",
+ " display(HTML(\n",
+ " data=\"\"\"\n",
+ " <video alt=\"test\" controls>\n",
+ " <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
+ " </video>\n",
+ " \"\"\".format(encoded.decode('ascii'))\n",
+ " ))\n",
+ "\n",
+ "ipython_show_video(glob.glob(\"/content/CartPole-v1/*.mp4\")[0])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "31e6af84-489e-4665-919e-8234462c1f0a",
+ "metadata": {
+ "id": "31e6af84-489e-4665-919e-8234462c1f0a"
+ },
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "collapsed_sections": [],
+ "name": "drl(2).ipynb",
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.7"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/DRL/FIDLE_rl_baselines_zoo.ipynb b/DRL/FIDLE_rl_baselines_zoo.ipynb
new file mode 100755
index 0000000000000000000000000000000000000000..54366632efe6f008bf8e59773a0dbd25492a15c4
--- /dev/null
+++ b/DRL/FIDLE_rl_baselines_zoo.ipynb
@@ -0,0 +1,546 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "XJy9QoDC7XA7"
+ },
+ "source": [
+ "<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n",
+ "\n",
+ "# <!-- TITLE --> [DRL2] - RL Baselines3 Zoo: Training in Colab\n",
+ "<!-- DESC --> Demo of Stable baseline3 with Colab\n",
+ "<!-- AUTHOR : Nathan Cassereau (IDRIS) and Bertrand Cabot (IDRIS) -->\n",
+ "\n",
+ "\n",
+ "Demo of Stable baseline3 adapted By Nathan Cassereau (IDRIS) and Bertrand Cabot (IDRIS)\n",
+ "\n",
+ "\n",
+ "Github Repo: [https://github.com/DLR-RM/rl-baselines3-zoo](https://github.com/DLR-RM/rl-baselines3-zoo)\n",
+ "\n",
+ "Stable-Baselines3 Repo: [https://github.com/DLR-RM/rl-baselines3-zoo](https://github.com/DLR-RM/stable-baselines3)\n",
+ "\n",
+ "\n",
+ "# Install Dependencies\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "AXVDDlTn02M9"
+ },
+ "outputs": [],
+ "source": [
+ "!apt-get install swig cmake ffmpeg freeglut3-dev xvfb"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "f1s9_3b1Tsq9"
+ },
+ "outputs": [],
+ "source": [
+ "!apt-get install -y \\\n",
+ " libgl1-mesa-dev \\\n",
+ " libgl1-mesa-glx \\\n",
+ " libglew-dev \\\n",
+ " libosmesa6-dev \\\n",
+ " software-properties-common\n",
+ "\n",
+ "!apt-get install -y patchelf"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kDjF3qRg7oGH"
+ },
+ "source": [
+ "## Clone RL Baselines3 Zoo Repo"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "SCjGikdT1DFy"
+ },
+ "outputs": [],
+ "source": [
+ "!git clone --recursive https://github.com/DLR-RM/rl-baselines3-zoo"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "REMQlh-ezyVt"
+ },
+ "outputs": [],
+ "source": [
+ "%cd /content/rl-baselines3-zoo/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5tmD_QTBqTMb"
+ },
+ "source": [
+ "### Install pip dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "OWIDzgJTqShY"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -r requirements.txt"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "VlOD-cImRMwW"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install free-mujoco-py"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kYPotFDF0Noa"
+ },
+ "source": [
+ "## Pretrained model\n",
+ "\n",
+ "gym environments: https://gym.openai.com/envs/"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "kgOp2XIklaf2"
+ },
+ "outputs": [],
+ "source": [
+ "%cd /content/rl-baselines3-zoo/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xVm9QPNVwKXN"
+ },
+ "source": [
+ "### Record a Video"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "MPyfQxD5z26J"
+ },
+ "outputs": [],
+ "source": [
+ "# Set up display; otherwise rendering will fail\n",
+ "import os\n",
+ "os.system(\"Xvfb :1 -screen 0 1024x768x24 &\")\n",
+ "os.environ['DISPLAY'] = ':1'"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZC3OTfpf8CXu"
+ },
+ "outputs": [],
+ "source": [
+ "import base64\n",
+ "from pathlib import Path\n",
+ "\n",
+ "from IPython import display as ipythondisplay\n",
+ "\n",
+ "def show_videos(video_path='', prefix=''):\n",
+ " \"\"\"\n",
+ " Taken from https://github.com/eleurent/highway-env\n",
+ "\n",
+ " :param video_path: (str) Path to the folder containing videos\n",
+ " :param prefix: (str) Filter the video, showing only the only starting with this prefix\n",
+ " \"\"\"\n",
+ " html = []\n",
+ " for mp4 in Path(video_path).glob(\"**/*{}*.mp4\".format(prefix)):\n",
+ " video_b64 = base64.b64encode(mp4.read_bytes())\n",
+ " html.append('''{} <br> <video alt=\"{}\" autoplay \n",
+ " loop controls style=\"height: 400px;\">\n",
+ " <source src=\"data:video/mp4;base64,{}\" type=\"video/mp4\" />\n",
+ " </video>'''.format(mp4, mp4, video_b64.decode('ascii')))\n",
+ " ipythondisplay.display(ipythondisplay.HTML(data=\"<br>\".join(html)))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LW-7EWA50550"
+ },
+ "source": [
+ "### Discrete environments"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ArN0B6YX0m4z"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/all_plots.py -a dqn qrdqn a2c ppo --env PongNoFrameskip-v4 -f rl-trained-agents/"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "5zmVA6JADKvV"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/plot_train.py -a dqn -e PongNoFrameskip-v4 -f rl-trained-agents/ -x time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "BkcI91hABKUm"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/plot_train.py -a qrdqn -e PongNoFrameskip-v4 -f rl-trained-agents/ -x time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "E5s7UHn2DmqQ"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/plot_train.py -a a2c -e PongNoFrameskip-v4 -f rl-trained-agents/ -x time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "0aP2A-NqCQSC"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/plot_train.py -a ppo -e PongNoFrameskip-v4 -f rl-trained-agents/ -x time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "itP336W6HdeC"
+ },
+ "outputs": [],
+ "source": [
+ "!python enjoy.py --algo dqn --env PongNoFrameskip-v4 --no-render --n-timesteps 5000"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ip3AauLzwNGP"
+ },
+ "outputs": [],
+ "source": [
+ "!python -m utils.record_video --algo dqn --env PongNoFrameskip-v4"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "oKOjFuwK9HI0"
+ },
+ "outputs": [],
+ "source": [
+ "show_videos(video_path='rl-trained-agents/dqn', prefix='PongNoFrameskip-v4')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8kD_C440-xvw"
+ },
+ "source": [
+ "### Continuous environments"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "2yJ0eOyjLM5O"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/all_plots.py -a ppo trpo sac td3 tqc --env Ant-v3 -f rl-trained-agents/"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "iQAJmndtNCvN"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/plot_train.py -a ppo -e Ant-v3 -f rl-trained-agents/ -x time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "O1L_ze67NDf9"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/plot_train.py -a trpo -e Ant-v3 -f rl-trained-agents/ -x time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "FPn0iySFNzCQ"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/plot_train.py -a tqc -e Ant-v3 -f rl-trained-agents/ -x time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "3vDHeNNTNwsT"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/plot_train.py -a td3 -e Ant-v3 -f rl-trained-agents/ -x time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "dQDNhdlgOzFs"
+ },
+ "outputs": [],
+ "source": [
+ "%run scripts/plot_train.py -a sac -e Ant-v3 -f rl-trained-agents/ -x time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "s9aT46m8QXzt"
+ },
+ "outputs": [],
+ "source": [
+ "!python enjoy.py --algo td3 --env Ant-v3 --no-render --n-timesteps 5000"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "zx7gZ23UQYQ4"
+ },
+ "outputs": [],
+ "source": [
+ "!python -m utils.record_video --algo td3 --env Ant-v3"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "yvETH_M_Unw3"
+ },
+ "outputs": [],
+ "source": [
+ "show_videos(video_path='rl-trained-agents/td3', prefix='Ant-v3')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6gJ-pAbF7zRZ"
+ },
+ "source": [
+ "## Train an RL Agent\n",
+ "\n",
+ "\n",
+ "The train agent can be found in the `logs/` folder.\n",
+ "\n",
+ "Here we will train A2C on CartPole-v1 environment for 100 000 steps. \n",
+ "\n",
+ "\n",
+ "To train it on Pong (Atari), you just have to pass `--env PongNoFrameskip-v4`\n",
+ "\n",
+ "Note: You need to update `hyperparams/algo.yml` to support new environments. You can access it in the side panel of Google Colab. (see https://stackoverflow.com/questions/46986398/import-data-into-google-colaboratory)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "background_save": true
+ },
+ "id": "9bIR_N7R11XI"
+ },
+ "outputs": [],
+ "source": [
+ "!python train.py --algo dqn --env PongNoFrameskip-v4 --n-timesteps 1000000"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-fHBq73665yD"
+ },
+ "source": [
+ "#### Evaluate trained agent\n",
+ "\n",
+ "\n",
+ "You can remove the `--folder logs/` to evaluate pretrained agent."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Bw8YuEgU6bT3"
+ },
+ "outputs": [],
+ "source": [
+ "!python enjoy.py --algo dqn --env PongNoFrameskip-v4 --no-render --n-timesteps 5000 --folder logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "w5Il2J0VHPLC"
+ },
+ "source": [
+ "#### Tune Hyperparameters\n",
+ "\n",
+ "We use [Optuna](https://optuna.org/) for optimizing the hyperparameters.\n",
+ "\n",
+ "Tune the hyperparameters for PPO, using a tpe sampler and median pruner, 2 parallels jobs,\n",
+ "with a budget of 1000 trials and a maximum of 50000 steps"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "w2sC22eGHTH-"
+ },
+ "outputs": [],
+ "source": [
+ "#!python train.py --algo dqn --env PongNoFrameskip-v4 -n 5000 -optimize --n-trials 10 --n-jobs 5 --sampler tpe --pruner median"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "qBuUfnzI8DN6"
+ },
+ "source": [
+ "### Display the video"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "RjdpP0HE8D2p"
+ },
+ "source": [
+ "### Continue Training\n",
+ "\n",
+ "Here, we will continue training of the previous model"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "zgMZQJJF6u1C"
+ },
+ "outputs": [],
+ "source": [
+ "#!python train.py --algo dqn --env PongNoFrameskip-v4 --n-timesteps 50000 -i logs/dqn/PongNoFrameskip-v4_1/PongNoFrameskip-v4.zip"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "GSaoyiAE8cVj"
+ },
+ "outputs": [],
+ "source": [
+ "#!python enjoy.py --algo dqn --env PongNoFrameskip-v4 --no-render --n-timesteps 1000 --folder logs/"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "SKglp1awG6c6"
+ },
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "collapsed_sections": [],
+ "name": "FIDLE rl-baselines-zoo.ipynb",
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/README.ipynb b/README.ipynb
index 9e0dd8b336ef007f626025a77467a3602a75cab2..08c5a9a7816a238450e823f383a8c9401f649268 100644
--- a/README.ipynb
+++ b/README.ipynb
@@ -3,13 +3,13 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "7f8c4a8a",
+ "id": "9e5009a6",
"metadata": {
"execution": {
- "iopub.execute_input": "2022-03-08T10:11:20.520072Z",
- "iopub.status.busy": "2022-03-08T10:11:20.519675Z",
- "iopub.status.idle": "2022-03-08T10:11:20.606245Z",
- "shell.execute_reply": "2022-03-08T10:11:20.606632Z"
+ "iopub.execute_input": "2022-03-29T20:08:36.649716Z",
+ "iopub.status.busy": "2022-03-29T20:08:36.646330Z",
+ "iopub.status.idle": "2022-03-29T20:08:36.658519Z",
+ "shell.execute_reply": "2022-03-29T20:08:36.658045Z"
},
"jupyter": {
"source_hidden": true
@@ -52,7 +52,7 @@
"[<img width=\"200px\" style=\"vertical-align:middle\" src=\"fidle/img/00-Mail_contact.svg\"></img>](#top)\n",
"\n",
"Current Version : <!-- VERSION_BEGIN -->\n",
- "**2.0.34**\n",
+ "**2.0.35**\n",
"<!-- VERSION_END -->\n",
"\n",
"\n",
@@ -170,8 +170,8 @@
"Episode : 3 Clustered dataset verification and testing of our datagenerator\n",
"- **[VAE8](VAE/08-VAE-with-CelebA.ipynb)** - [Training session for our VAE](VAE/08-VAE-with-CelebA.ipynb) \n",
"Episode 4 : Training with our clustered datasets in notebook or batch mode\n",
- "- **[VAE9](VAE/10-VAE-with-CelebA-post.ipynb)** - [Data generation from latent space](VAE/10-VAE-with-CelebA-post.ipynb) \n",
- "Episode 5 : Exploring latent space to generate new data\n",
+ "- **[VAE9](VAE/09-VAE-with-CelebA-192x160.ipynb)** - [Training session for our VAE with 192x160 images](VAE/09-VAE-with-CelebA-192x160.ipynb) \n",
+ "Episode 4 : Training with our clustered datasets in notebook or batch mode\n",
"- **[VAE10](VAE/batch_slurm.sh)** - [SLURM batch script](VAE/batch_slurm.sh) \n",
"Bash script for SLURM batch submission of VAE8 notebooks \n",
"\n",
@@ -181,6 +181,12 @@
"- **[SHEEP2](DCGAN/02-WGANGP-Draw-me-a-sheep.ipynb)** - [A WGAN-GP to Draw a Sheep](DCGAN/02-WGANGP-Draw-me-a-sheep.ipynb) \n",
"Episode 2 : Draw me a sheep, revisited with a WGAN-GP\n",
"\n",
+ "### Deep Reinforcement Learning (DRL)\n",
+ "- **[DRL1](DRL/FIDLE_DQNfromScratch.ipynb)** - [Solving CartPole with DQN](DRL/FIDLE_DQNfromScratch.ipynb) \n",
+ "Using a a Deep Q-Network to play CartPole - an inverted pendulum problem (PyTorch)\n",
+ "- **[DRL2](DRL/FIDLE_rl_baselines_zoo.ipynb)** - [RL Baselines3 Zoo: Training in Colab](DRL/FIDLE_rl_baselines_zoo.ipynb) \n",
+ "Demo of Stable baseline3 with Colab\n",
+ "\n",
"### Miscellaneous\n",
"- **[ACTF1](Misc/Activation-Functions.ipynb)** - [Activation functions](Misc/Activation-Functions.ipynb) \n",
"Some activation functions, with their derivatives.\n",
diff --git a/README.md b/README.md
index 68f097aeee55cf26b882560ddeee67298e9ef742..2dc61b106b27ef6c545c654393eb154173ce6a55 100644
--- a/README.md
+++ b/README.md
@@ -31,7 +31,7 @@ For more information, you can contact us at :
[<img width="200px" style="vertical-align:middle" src="fidle/img/00-Mail_contact.svg"></img>](#top)
Current Version : <!-- VERSION_BEGIN -->
-**2.0.34**
+**2.0.35**
<!-- VERSION_END -->
@@ -149,8 +149,8 @@ Episode 2 : Analysis of the CelebA dataset and creation of an clustered and usab
Episode : 3 Clustered dataset verification and testing of our datagenerator
- **[VAE8](VAE/08-VAE-with-CelebA.ipynb)** - [Training session for our VAE](VAE/08-VAE-with-CelebA.ipynb)
Episode 4 : Training with our clustered datasets in notebook or batch mode
-- **[VAE9](VAE/10-VAE-with-CelebA-post.ipynb)** - [Data generation from latent space](VAE/10-VAE-with-CelebA-post.ipynb)
-Episode 5 : Exploring latent space to generate new data
+- **[VAE9](VAE/09-VAE-with-CelebA-192x160.ipynb)** - [Training session for our VAE with 192x160 images](VAE/09-VAE-with-CelebA-192x160.ipynb)
+Episode 4 : Training with our clustered datasets in notebook or batch mode
- **[VAE10](VAE/batch_slurm.sh)** - [SLURM batch script](VAE/batch_slurm.sh)
Bash script for SLURM batch submission of VAE8 notebooks
@@ -160,6 +160,12 @@ Episode 1 : Draw me a sheep, revisited with a DCGAN
- **[SHEEP2](DCGAN/02-WGANGP-Draw-me-a-sheep.ipynb)** - [A WGAN-GP to Draw a Sheep](DCGAN/02-WGANGP-Draw-me-a-sheep.ipynb)
Episode 2 : Draw me a sheep, revisited with a WGAN-GP
+### Deep Reinforcement Learning (DRL)
+- **[DRL1](DRL/FIDLE_DQNfromScratch.ipynb)** - [Solving CartPole with DQN](DRL/FIDLE_DQNfromScratch.ipynb)
+Using a a Deep Q-Network to play CartPole - an inverted pendulum problem (PyTorch)
+- **[DRL2](DRL/FIDLE_rl_baselines_zoo.ipynb)** - [RL Baselines3 Zoo: Training in Colab](DRL/FIDLE_rl_baselines_zoo.ipynb)
+Demo of Stable baseline3 with Colab
+
### Miscellaneous
- **[ACTF1](Misc/Activation-Functions.ipynb)** - [Activation functions](Misc/Activation-Functions.ipynb)
Some activation functions, with their derivatives.
diff --git a/fidle/01-update-index.ipynb b/fidle/01-update-index.ipynb
index cd919a95e13ab40e0ba424f26b1e110adddde584..8994f52eb52cca8004850afb21472ae580fc7764 100644
--- a/fidle/01-update-index.ipynb
+++ b/fidle/01-update-index.ipynb
@@ -67,6 +67,7 @@
" 'AE':'Unsupervised learning with an autoencoder neural network (AE)',\n",
" 'VAE':'Generative network with Variational Autoencoder (VAE)',\n",
" 'DCGAN':'Generative Adversarial Networks (GANs)',\n",
+ " 'DRL':'Deep Reinforcement Learning (DRL)',\n",
" 'Misc':'Miscellaneous'\n",
" }"
]
diff --git a/fidle/ci/default.yml b/fidle/ci/default.yml
index 9c7ebc5cadf8ef5e072824ec90284563b2bef286..60267f2d385bfb5ff0e99540a41e0ba0791eb333 100644
--- a/fidle/ci/default.yml
+++ b/fidle/ci/default.yml
@@ -413,12 +413,18 @@ Nb_VAE8:
Nb_VAE9:
notebook_id: VAE9
notebook_dir: VAE
- notebook_src: 10-VAE-with-CelebA-post.ipynb
+ notebook_src: 09-VAE-with-CelebA-192x160.ipynb
notebook_tag: default
overrides:
run_dir: default
+ scale: default
image_size: default
enhanced_dir: default
+ latent_dim: default
+ loss_weights: default
+ batch_size: default
+ epochs: default
+ fit_verbosity: default
Nb_VAE10:
notebook_id: VAE10
notebook_dir: VAE
@@ -450,6 +456,16 @@ Nb_SHEEP2:
batch_size: default
num_img: default
fit_verbosity: default
+Nb_DRL1:
+ notebook_id: DRL1
+ notebook_dir: DRL
+ notebook_src: FIDLE_DQNfromScratch.ipynb
+ notebook_tag: default
+Nb_DRL2:
+ notebook_id: DRL2
+ notebook_dir: DRL
+ notebook_src: FIDLE_rl_baselines_zoo.ipynb
+ notebook_tag: default
Nb_ACTF1:
notebook_id: ACTF1
notebook_dir: Misc
diff --git a/fidle/config.py b/fidle/config.py
index 8423d93fcbf8485ea59f589d4ced15e5918cb853..637d15da657b479dbe89aa4f73e0ff8cd90d3cf3 100644
--- a/fidle/config.py
+++ b/fidle/config.py
@@ -14,7 +14,7 @@
# ---- Version -----------------------------------------------------
#
-VERSION = '2.0.34'
+VERSION = '2.0.35'
# ---- Default notebook name ---------------------------------------
#
diff --git a/fidle/logs/catalog.json b/fidle/logs/catalog.json
index ab65f18c8a0b0eab6917580734ede7adeed09421..f7e7c8b39f4389d7c4dbe23405dc407b1985c2ff 100644
--- a/fidle/logs/catalog.json
+++ b/fidle/logs/catalog.json
@@ -531,13 +531,20 @@
"VAE9": {
"id": "VAE9",
"dirname": "VAE",
- "basename": "10-VAE-with-CelebA-post.ipynb",
- "title": "Data generation from latent space",
- "description": "Episode 5 : Exploring latent space to generate new data",
+ "basename": "09-VAE-with-CelebA-192x160.ipynb",
+ "title": "Training session for our VAE with 192x160 images",
+ "description": "Episode 4 : Training with our clustered datasets in notebook or batch mode",
"overrides": [
"run_dir",
+ "scale",
"image_size",
- "enhanced_dir"
+ "enhanced_dir",
+ "latent_dim",
+ "loss_weights",
+ "batch_size",
+ "epochs",
+ "fit_verbosity",
+ "run_dir"
]
},
"VAE10": {
@@ -582,6 +589,22 @@
"run_dir"
]
},
+ "DRL1": {
+ "id": "DRL1",
+ "dirname": "DRL",
+ "basename": "FIDLE_DQNfromScratch.ipynb",
+ "title": "Solving CartPole with DQN",
+ "description": "Using a a Deep Q-Network to play CartPole - an inverted pendulum problem (PyTorch)",
+ "overrides": []
+ },
+ "DRL2": {
+ "id": "DRL2",
+ "dirname": "DRL",
+ "basename": "FIDLE_rl_baselines_zoo.ipynb",
+ "title": "RL Baselines3 Zoo: Training in Colab",
+ "description": "Demo of Stable baseline3 with Colab",
+ "overrides": []
+ },
"ACTF1": {
"id": "ACTF1",
"dirname": "Misc",