03-Using-Pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "51be1de8",
   "metadata": {},
   "source": [
    "<img width=\"800px\" src=\"../fidle/img/header.svg\"></img>\n",
    "\n",
    "# <!-- TITLE --> [PYTORCH1] - Practical Lab : PyTorch\n",
    "<!-- DESC --> PyTorch est l'un des principaux framework utilisé dans le Deep Learning\n",
    "<!-- AUTHOR : Kamel Guerda (CNRS/IDRIS) -->\n",
    "\n",
    "## Objectives :\n",
    " - Understand PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1959d3d5-388e-4c43-8318-342f08e6b024",
   "metadata": {
    "tags": []
   },
   "source": [
    "## **Introduction**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6da1305-551a-4549-abed-641415823a33",
   "metadata": {},
   "source": [
    "**PyTorch** is an open-source machine learning library developed by Facebook's AI Research lab. It offers an imperative and dynamic computational model, making it particularly easy and intuitive for researchers. Its primary feature is the tensor, a multi-dimensional array similar to NumPy's ndarray, but with GPU acceleration."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54c79dfb-a061-4b72-afe3-c97c28071e5c",
   "metadata": {
    "tags": []
   },
   "source": [
    "### **Installation and usage**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20852981-c289-4c4e-8099-2c5efef58e3b",
   "metadata": {},
   "source": [
    "Whether you're working on the supercomputer Jean Zay or your own machine, getting your environment ready is the first step. Here's how to proceed:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a88f32bd-37f6-4e99-97e0-62283a146a1f",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### **On Jean Zay**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8421a9f0-130d-40ef-8a7a-066bf9147066",
   "metadata": {},
   "source": [
    "For those accessing the Jean Zay supercomputer (you should already be at step 3):\n",
    "\n",
    "1. **Access JupyterHub**: Go to [https://jupyterhub.idris.fr](https://jupyterhub.idris.fr). The login credentials are the same as those used to access the Jean Zay machine. Ensure your IP address is whitelisted (add a new IP via the account management form if needed).\n",
    "2. **Create a JupyterLab Instance**: Choose to create the instance either on a frontend node (e.g., for internet access) or on a compute node by reserving resources via Slurm. Select the appropriate options such as workspace, allocated resources, billing, etc.\n",
    "3. **Choose the Kernel**: IDRIS provides kernels based on modules installed on Jean Zay. This includes various versions of Python, Tensorflow, and PyTorch. Create a new notebook with the desired kernel through the launcher or change the kernel on an existing notebook by clicking the kernel name at the top right of the screen.\n",
    "4. For advanced features like Tensorboard, MLFlow, custom kernel creation, etc., refer to the [JupyterHub technical documentation](https://jupyterhub.idris.fr/services/documentation/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a168594c-cf18-4ed8-babf-242b56b3e0b7",
   "metadata": {
    "tags": []
   },
   "source": [
    "> **Task:** Verifying Your Kernel in the upper top corner\n",
    ">    - In JupyterLab, at the top right of your notebook, you should see the name of your current kernel.\n",
    ">    - Ensure it matches \"PyTorch 2.0\" or a similar name indicating the PyTorch version.\n",
    ">    - If it doesn't, click on the kernel name and select the appropriate kernel from the list.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0aaadeee-5115-48d0-aa57-20a0a63d5054",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### **Elsewhere**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d34951e-1b7b-4776-9449-eff57a9385f4",
   "metadata": {},
   "source": [
    "\n",
    "For users on other platforms:\n",
    "\n",
    "1. Install PyTorch by following the official [installation guide](https://pytorch.org/get-started/locally/).\n",
    "2. If you have a GPU, ensure you've installed the necessary CUDA toolkit and cuDNN libraries.\n",
    "3. Launch your preferred Python environment, whether it's Jupyter notebook, an IDE like PyCharm, or just the terminal.\n",
    "\n",
    "Once your setup is complete, you're ready to dive in. Let's explore the fascinating world of deep learning!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7552d5ac-eb8c-48e0-9e61-3b056d560f7b",
   "metadata": {
    "tags": []
   },
   "source": [
    "### **Version**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "272e492f-35c5-4293-b504-8e8632da1b73",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Importing PyTorch\n",
    "import torch\n",
    "\n",
    "# TODO: Print the version of PyTorch being used\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fdbe225-4e06-4ad0-abca-4325457dc0e1",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "To print the version of PyTorch you're using, you can access the <code>__version__</code> attribute of the <code>torch</code> module.    \n",
    "    \n",
    "```python\n",
    "print(torch.__version__)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72752068-02fe-4e44-8c27-40e8f66680c9",
   "metadata": {
    "tags": []
   },
   "source": [
    "**Why PyTorch 2.0 is a Game-Changer**\n",
    "\n",
    "PyTorch 2.0 represents a major step in the evolution of this popular deep learning library. As part of the transition to the 2-series, let's highlight some reasons why this version is pivotal:\n",
    "\n",
    "1. **Performance**: With PyTorch 2.0, performance has been supercharged at the compiler level, offering faster execution and support for Dynamic Shapes and Distributed systems.\n",
    "  \n",
    "2. **torch.compile**: This introduces a more Pythonic approach, moving some parts of PyTorch from C++ back to Python. Notably, across a test set of 163 open-source models, the use of `torch.compile` resulted in a 43% speed increase during training on an NVIDIA A100 GPU.\n",
    "\n",
    "3. **Innovative Technologies**: Technologies like TorchDynamo and TorchInductor, both written in Python, make PyTorch more flexible and developer-friendly.\n",
    "  \n",
    "4. **Staying Pythonic**: PyTorch 2.0 emphasizes Python-centric development, reducing barriers for developers and vendors.\n",
    "\n",
    "As we progress in this lab, we'll dive deeper into some of these features, giving you hands-on experience with the power and flexibility of PyTorch 2.0.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc215c02-1f16-48be-88f9-5080fd2be9ed",
   "metadata": {
    "tags": []
   },
   "source": [
    "## **Pytorch Fundamentals**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcd7f0fc-a714-495e-9307-e48964abd85b",
   "metadata": {
    "tags": []
   },
   "source": [
    "### **Tensors**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e185bf6-3d3c-4a43-b425-e6aa3da5d5dd",
   "metadata": {
    "tags": []
   },
   "source": [
    "A **tensor** is a generalization of vectors and matrices and is easily understood as a multi-dimensional array. In the context of PyTorch:\n",
    "- A 0-dimensional tensor is a scalar (a single number).\n",
    "- A 1-dimensional tensor is a vector.\n",
    "- A 2-dimensional tensor is a matrix.\n",
    "- ... and so on for higher dimensions.\n",
    "\n",
    "Tensors are fundamental to PyTorch not just as data containers but also for their compatibility with GPU acceleration, making operations on them extremely fast. This acceleration is vital for training large neural networks.\n",
    "\n",
    "Let's start our journey with tensors by examining how PyTorch handles scalars."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa90e399-3955-4417-a4a3-c0c812ebb1d9",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### **Scalars in PyTorch**\n",
    "\n",
    "### Scalars in PyTorch\n",
    "\n",
    "A scalar, being a 0-dimensional tensor, is simply a single number. While it might seem trivial, understanding scalars in PyTorch lays the foundation for grasping more complex tensor structures. Familiarize yourself with the `torch.tensor()` function from the [official documentation](https://pytorch.org/docs/stable/generated/torch.tensor.html) before proceeding.\n",
    "\n",
    "> **Task**: Create a scalar tensor in PyTorch and examine its properties.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b6db1841-0fab-4df0-b699-058d5a477ca6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (2309926818.py, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  Cell \u001b[0;32mIn[2], line 2\u001b[0;36m\u001b[0m\n\u001b[0;31m    scalar_tensor = # Your code here\u001b[0m\n\u001b[0m                    ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# TODO: Create a scalar tensor with the value 7.5\n",
    "scalar_tensor = # Your code here\n",
    "\n",
    "# Print the scalar tensor\n",
    "print(\"Scalar Tensor:\", scalar_tensor)\n",
    "\n",
    "# TODO: Print its dimension, shape, and type\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9bc265c-9a7f-4588-8586-562b390d63d9",
   "metadata": {
    "tags": []
   },
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "To create a scalar tensor, use the <code>torch.tensor()</code> function. To retrieve its dimension, shape, and type, you can use the <code>.dim()</code>, <code>.shape</code>, and <code>.dtype</code> attributes respectively. \n",
    "\n",
    "Here's how you can achieve that:\n",
    "\n",
    "```python\n",
    "scalar_tensor = torch.tensor(7.5)\n",
    "print(\"Scalar Tensor:\", scalar_tensor)\n",
    "print(\"Dimension:\", scalar_tensor.dim())\n",
    "print(\"Shape:\", scalar_tensor.shape)\n",
    "print(\"Type:\", scalar_tensor.dtype)\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc240c26-5866-4080-bbb9-d5cde1500300",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### **Vectors in PyTorch**\n",
    "\n",
    "A vector in PyTorch is a 1-dimensional tensor. It's essentially a list of numbers that can represent anything from a sequence of data points to the weights of a neural network layer.\n",
    "\n",
    "In this section, we'll see how to create and manipulate vectors using PyTorch. We'll also look at some basic operations you can perform on them.\n",
    "\n",
    "> **Task**: Create a 1-dimensional tensor (vector) with values `[1.5, 2.3, 3.1, 4.8, 5.2]` and print its dimension, shape, and type.\n",
    "\n",
    "Start by referring to the `torch.tensor()` function in the [official documentation](https://pytorch.org/docs/stable/generated/torch.tensor.html) to understand how to create tensors of varying dimensions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e9503b49-38d1-45d9-910f-761da82cfbd0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (138343520.py, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  Cell \u001b[0;32mIn[3], line 2\u001b[0;36m\u001b[0m\n\u001b[0;31m    vector_tensor = # Your code here\u001b[0m\n\u001b[0m                    ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# TODO: Create a 1-dimensional tensor (vector) with values [1.5, 2.3, 3.1, 4.8, 5.2]\n",
    "vector_tensor = # Your code here\n",
    "\n",
    "# Print the vector tensor\n",
    "print(\"Vector Tensor:\", vector_tensor)\n",
    "\n",
    "# TODO: Print its dimension, shape, and type\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13252d1f-004f-42e0-aec9-56322b43ab72",
   "metadata": {
    "tags": []
   },
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "Creating a 1-dimensional tensor is similar to creating a scalar. Instead of a single number, you pass a list of numbers to the <code>torch.tensor()</code> function. The <code>.dim()</code>, <code>.shape</code>, and <code>.dtype</code> attributes will help you retrieve its properties.\n",
    "\n",
    "```python\n",
    "vector_tensor = torch.tensor([1.5, 2.3, 3.1, 4.8, 5.2])\n",
    "print(\"Vector Tensor:\", vector_tensor)\n",
    "print(\"Dimension:\", vector_tensor.dim())\n",
    "print(\"Shape:\", vector_tensor.shape)\n",
    "print(\"Type:\", vector_tensor.dtype)\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bfc47a8-e99d-4683-ac36-287f35a76fd0",
   "metadata": {},
   "source": [
    "#### **Vector Operations**\n",
    "\n",
    "Vectors are not just static entities; we often perform various operations on them, especially in the context of neural networks. This includes addition, subtraction, scalar multiplication, dot products, etc.\n",
    "\n",
    "> **Task**: Using the previously defined `vector_tensor`, perform the following operations:\n",
    "1. Add 5 to all the elements of the vector.\n",
    "2. Multiply all the elements of the vector by 2.\n",
    "3. Compute the dot product of the vector with itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "86182e1c-5491-4743-a7c8-10b9effd8194",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (184231995.py, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  Cell \u001b[0;32mIn[4], line 2\u001b[0;36m\u001b[0m\n\u001b[0;31m    vector_added = # Your code here\u001b[0m\n\u001b[0m                   ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# TODO: Add 5 to all elements\n",
    "vector_added = # Your code here\n",
    "\n",
    "# TODO: Multiply all elements by 2\n",
    "vector_multiplied = # Your code here\n",
    "\n",
    "# TODO: Compute the dot product with itself\n",
    "dot_product = # Your code here\n",
    "\n",
    "# Print the results\n",
    "print(\"Vector after addition:\", vector_added)\n",
    "print(\"Vector after multiplication:\", vector_multiplied)\n",
    "print(\"Dot Product:\", dot_product)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75773a02-3ab4-4325-99fb-7a742e997f21",
   "metadata": {
    "tags": []
   },
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "PyTorch tensors support regular arithmetic operations. For the dot product, you can use the <code>torch.dot()</code> function.\n",
    "\n",
    "```python\n",
    "\n",
    "vector_added = vector_tensor + 5\n",
    "vector_multiplied = vector_tensor * 2\n",
    "dot_product = torch.dot(vector_tensor, vector_tensor)\n",
    "\n",
    "print(\"Vector after addition:\", vector_added)\n",
    "print(\"Vector after multiplication:\", vector_multiplied)\n",
    "print(\"Dot Product:\", dot_product)\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b4766ba-ef9a-4f24-ba43-7358097a7b61",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### **Matrices in PyTorch**\n",
    "\n",
    "A matrix in PyTorch is represented as a 2D tensor. Just as vectors are generalizations of scalars, matrices are generalizations of vectors, providing an additional dimension. Matrices are crucial for a range of operations in deep learning, including representing datasets, transformations, and more.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ec7544d-ef87-4773-88d8-cee731d1c43c",
   "metadata": {
    "tags": []
   },
   "source": [
    "##### **Creating Matrices**\n",
    "\n",
    "Before diving into manual matrix creation, it's beneficial to know some utility functions PyTorch provides:\n",
    "\n",
    "- `torch.rand()`: Generates a matrix with random values between 0 and 1.\n",
    "- `torch.eye()`: Creates an identity matrix.\n",
    "- `torch.zeros()`: Generates a matrix filled with zeros.\n",
    "- `torch.ones()`: Generates a matrix filled with ones.\n",
    "\n",
    "You can explore more about these functions in the [official documentation](https://pytorch.org/docs/stable/tensors.html).\n",
    "\n",
    "> **Task**: Using the above functions, create the following matrices:\n",
    "> 1. A 3x3 matrix with random values.\n",
    "> 2. A 5x5 identity matrix.\n",
    "> 3. A 2x4 matrix filled with zeros.\n",
    "> 4. A 4x2 matrix filled with ones.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5014b564-6bf5-4f00-a513-578ca72d94a8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for creating the matrices goes here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86b2708c-45c6-4b2c-b526-41491fcafa08",
   "metadata": {
    "tags": []
   },
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "\n",
    "To create these matrices, make use of the following functions:\n",
    "\n",
    "1. `torch.rand(size)`: Use this function and specify the size as `(3, 3)` to create a 3x3 matrix with random values.\n",
    "2. `torch.eye(n, m)`: Use this to generate an identity matrix. For a square matrix like 5x5, n and m would both be 5.\n",
    "3. `torch.zeros(m, n)`: For a 2x4 matrix filled with zeros, specify m=2 and n=4.\n",
    "4. `torch.ones(m, n)`: Similar to the `zeros` function but fills the matrix with ones.\n",
    "\n",
    "```python\n",
    "# 1. 3x3 matrix with random values\n",
    "random_matrix = torch.rand(3, 3)\n",
    "print(random_matrix)\n",
    "\n",
    "# 2. 5x5 identity matrix\n",
    "identity_matrix = torch.eye(5, 5)\n",
    "print(identity_matrix)\n",
    "\n",
    "# 3. 2x4 matrix filled with zeros\n",
    "zero_matrix = torch.zeros(2, 4)\n",
    "print(zero_matrix)\n",
    "\n",
    "# 4. 4x2 matrix filled with ones\n",
    "one_matrix = torch.ones(4, 2)\n",
    "print(one_matrix)\n",
    "```\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60ff5e51-699e-46a1-8cc7-1d5fc9a4d078",
   "metadata": {},
   "source": [
    "#### **Matrix Operations in PyTorch**\n",
    "\n",
    "Just like vectors, matrices can undergo a variety of operations. Some of the basic ones include matrix addition, subtraction, and multiplication. More advanced operations include matrix inversion, transposition, and determinant calculation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6bdb9d9-b299-4d63-b92f-7c4b8c32a1b7",
   "metadata": {
    "tags": []
   },
   "source": [
    "##### **Basic Matrix Operations**\n",
    "\n",
    "> **Task**: Perform the following operations on matrices:\n",
    "> 1. Create two 3x3 matrices with random values.\n",
    "> 2. Add the two matrices.\n",
    "> 3. Subtract the second matrix from the first one.\n",
    "> 4. Multiply the two matrices element-wise.\n",
    "\n",
    "Remember, for matrix multiplication that results in the dot product, you'd use `torch.mm` or `@`, but for element-wise multiplication, you use `*`.\n",
    "\n",
    "Here's the [official documentation](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.matmul) on matrix operations for your reference.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6be8c647-c455-4d3b-8a21-c4b7102ffa75",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for creating the matrices and performing the operations goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0020b26b-b2bb-4efa-9bf3-3f037acd050e",
   "metadata": {
    "tags": []
   },
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "\n",
    "Here's how you can perform the given matrix operations:\n",
    "\n",
    "```python\n",
    "# 1. Create two 3x3 matrices with random values\n",
    "matrix1 = torch.rand(3, 3)\n",
    "matrix2 = torch.rand(3, 3)\n",
    "print(\"Matrix 1:\\n\", matrix1)\n",
    "print(\"\\nMatrix 2:\\n\", matrix2)\n",
    "\n",
    "# 2. Add the two matrices\n",
    "sum_matrix = matrix1 + matrix2\n",
    "print(\"\\nSum of matrices:\\n\", sum_matrix)\n",
    "\n",
    "# 3. Subtract the second matrix from the first one\n",
    "difference_matrix = matrix1 - matrix2\n",
    "print(\"\\nDifference of matrices:\\n\", difference_matrix)\n",
    "\n",
    "# 4. Multiply the two matrices element-wise\n",
    "product_matrix = matrix1 * matrix2\n",
    "print(\"\\nElement-wise product of matrices:\\n\", product_matrix)\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07f57464-76e2-4670-8332-3fcec2e162bd",
   "metadata": {},
   "source": [
    "#### **Higher-Dimensional Tensors in PyTorch**\n",
    "\n",
    "While scalars, vectors, and matrices cover 0D, 1D, and 2D tensors respectively, in deep learning, especially in tasks like image processing, you often encounter tensors with more than two dimensions.\n",
    "\n",
    "For instance, a colored image is often represented as a 3D tensor: height x width x channels (e.g., RGB channels). A batch of such images would then be a 4D tensor: batch_size x height x width x channels.\n",
    "\n",
    "Let's get our hands dirty with some higher-dimensional tensors!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dd1fea7-d290-49fe-ac1f-5a8387e3d386",
   "metadata": {
    "tags": []
   },
   "source": [
    "##### **Creating a 3D Tensor**\n",
    "\n",
    "> **Task**: Create a 3D tensor representing 2 images of size 4x4 with 3 channels (like RGB) filled with random values.\n",
    "\n",
    "Use the `torch.rand` function, and remember to specify the dimensions correctly.\n",
    "\n",
    "Here's the [official documentation](https://pytorch.org/docs/stable/tensors.html#creation-ops) for tensor creation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7c8ac6e-f870-4b5d-ac2c-05be1d0cc9f1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for creating the 3D tensor goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe61750-a91f-428a-b4e2-7df0cc2a782b",
   "metadata": {
    "tags": []
   },
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "\n",
    "Creating a 3D tensor with the given specifications can be achieved using the `torch.rand` function. Here's how:\n",
    "\n",
    "```python\n",
    "# Create a 3D tensor representing 2 images of size 4x4 with 3 channels\n",
    "image_tensor = torch.rand(2, 4, 4, 3)\n",
    "print(image_tensor)\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cfbcaa0-a0f6-4869-ba94-65d4439a60ca",
   "metadata": {},
   "source": [
    "#### **Reshaping Tensors**\n",
    "\n",
    "In deep learning, we often need to reshape our tensors. For instance, an image represented as a 3D tensor might need to be reshaped into a 1D tensor before passing it through a fully connected layer. PyTorch provides methods to make this easy.\n",
    "\n",
    "The most commonly used method for reshaping tensors in PyTorch is the `view()` method. Another method that offers more flexibility (especially when you're unsure about the size of one dimension) is `reshape()`.\n",
    "\n",
    ">[Task]: Using the official documentation, find out how to use the [`view()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view) and [`reshape()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.reshape) methods. Create a 2x3 tensor using `torch.tensor()` and then reshape it into a 3x2 tensor.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6758ba7-aa35-42f0-87c1-86b88de64238",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create a 2x3 tensor\n",
    "\n",
    "# Reshape it into a 3x2 tensor\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fea31255-c2fe-47b2-b03b-c2b35953e05a",
   "metadata": {
    "tags": []
   },
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "To reshape a tensor using <code>view()</code> method:\n",
    "\n",
    "```python\n",
    "tensor = torch.tensor([[1, 2, 3], [4, 5, 6]])\n",
    "reshaped_tensor = tensor.view(3, 2)\n",
    "```\n",
    "<br>\n",
    "Alternatively, using the <code>reshape()</code> method:\n",
    "\n",
    "```python\n",
    "reshaped_tensor = tensor.reshape(3, 2)\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c580dbca-b75a-4b97-a24a-6a19c7cdf8d1",
   "metadata": {},
   "source": [
    "#### **Broadcasting**\n",
    "\n",
    "Broadcasting is a powerful feature in PyTorch that allows you to perform operations between tensors of different shapes. When possible, PyTorch will automatically reshape the tensors in a way that makes the operation valid. This can significantly reduce manual reshaping and is efficient in memory usage.\n",
    "\n",
    "However, it's essential to understand the rules and nuances of broadcasting to use it effectively and avoid unexpected behaviors.\n",
    "\n",
    ">[Task]: Given a tensor `A` of shape (4, 1) and another tensor `B` of shape (1, 4), use PyTorch operations to produce a result tensor of shape (4, 4). Check the [official documentation on broadcasting](https://pytorch.org/docs/stable/notes/broadcasting.html) for guidance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44566fb7-87ed-41ef-a86e-db32a1cf2179",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define tensor A of shape (4, 1) and tensor B of shape (1, 4)\n",
    "\n",
    "# Perform an operation to get a result tensor of shape (4, 4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2602f2c4-f507-4a9a-8e8d-dee5e95efc61",
   "metadata": {
    "tags": []
   },
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "You can simply use addition, subtraction, multiplication, or any other element-wise operation. When you do this operation, PyTorch will automatically broadcast the tensors to a compatible shape. For example:\n",
    "\n",
    "```python\n",
    "A = torch.tensor([[1], [2], [3], [4]])\n",
    "B = torch.tensor([[1, 2, 3, 4]])\n",
    "result = A * B\n",
    "print(result)\n",
    "```\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba2cc439-8ecc-4d92-b78f-39ef762678f8",
   "metadata": {
    "tags": []
   },
   "source": [
    "### **GPU Support with CUDA**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "575536c5-87a7-4781-8557-558627f14c0a",
   "metadata": {
    "tags": []
   },
   "source": [
    "PyTorch seamlessly supports operations on Graphics Processing Units (GPUs) through CUDA, an API developed by NVIDIA for their GPUs. If you have a compatible NVIDIA GPU on your machine, PyTorch can utilize it to speed up tensor operations which can be orders of magnitude faster than on a CPU.\n",
    "\n",
    "To verify if your PyTorch installation can use CUDA, you can check the attribute `torch.cuda.is_available()`. This returns `True` if CUDA is available and PyTorch can use GPUs, otherwise it returns `False`.\n",
    "\n",
    ">[Task]: Print whether CUDA support is available on your system. The [CUDA documentation](https://pytorch.org/docs/stable/cuda.html) might be useful for this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38e84bb7-5026-4262-8b78-b368c55a1450",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Check and print if CUDA is available\n",
    "cuda_available = None  # Replace None with the appropriate code\n",
    "print(\"CUDA available:\", cuda_availablez"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "646b5660-5131-4ce0-9592-0fd14608c6df",
   "metadata": {
    "tags": []
   },
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "\n",
    "To check if CUDA is available, you can utilize the torch.cuda.is_available() function.\n",
    "```python\n",
    "cuda_available = torch.cuda.is_available()\n",
    "print(\"CUDA available:\", cuda_available)\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86c8d7ed-0931-4874-bb27-e796ae1a1d7a",
   "metadata": {},
   "source": [
    "When developing deep learning models in PyTorch, it's a good habit to write device-agnostic code. This means your code can automatically use a GPU if available, or fall back to using the CPU if not. The `torch.device` object allows you to specify the device (either CPU or GPU) where you'd like your tensors to be allocated.\n",
    "\n",
    "To dynamically determine the device, a common pattern is to check `torch.cuda.is_available()`, and set the device accordingly. This is particularly useful when you want your code to be flexible, regardless of the underlying hardware.\n",
    "\n",
    ">[Task]: Define a `device` variable that is set to 'cuda:0' if CUDA is available and 'cpu' otherwise. Create a tensor on this device. The [documentation about torch.device](https://pytorch.org/docs/stable/tensor_attributes.html#torch-device) might be handy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91e05e75-03ad-44cb-9842-89e2017ee709",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the device\n",
    "device = None  # Replace None with the appropriate code\n",
    "\n",
    "# Create a tensor on the specified device\n",
    "tensor_on_device = torch.tensor([1, 2, 3, 4, 5], device=device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b80406b-b1cc-4831-a6ba-8e6385703755",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "\n",
    "To define the device variable dynamically:\n",
    "\n",
    "```python\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "```\n",
    "<br>\n",
    "After setting the device, you can create tensors on it directly using the device argument.\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "574a2192-cc09-4d2c-8f01-97b051b7ffc8",
   "metadata": {
    "tags": []
   },
   "source": [
    "### **Automatic Differentiation with Autograd**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f5406f6-e295-4f70-a815-9eef18352390",
   "metadata": {
    "tags": []
   },
   "source": [
    "PyTorch's `autograd` module provides the tools for automatically computing the gradients for tensors. This feature is a cornerstone for neural network training, as gradients are essential for optimization algorithms like gradient descent.\n",
    "\n",
    "When we create a tensor, `requires_grad` is set to `False` by default, meaning it won't track operations. However, if we set `requires_grad=True`, PyTorch will start to track all operations on the tensor.\n",
    "\n",
    "Let's start with a simple example:\n",
    "\n",
    ">**Task:** Create a tensor that holds a single value, let's say 2, and set `requires_grad=True`. Then, define a simple operation like squaring the tensor. Finally, inspect the resulting tensor. The [documentation for requires_grad](https://pytorch.org/docs/stable/autograd.html#torch.Tensor.requires_grad) might be handy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe63ab93-55be-434d-822f-8fd9cd727941",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# TODO: Create a tensor, perform a simple operation, and print its data and grad_fn separately.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa7ee20c-c2d6-4dcf-bb37-9eda580b5dc5",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "\n",
    "To create a tensor with requires_grad=True and square it:\n",
    "\n",
    "```python\n",
    "# TODO: Create a tensor, perform a simple operation, and print its data and grad_fn separately.\n",
    "x = torch.tensor([2.0], requires_grad=True)\n",
    "y = x ** 2\n",
    "print(\"Data:\", y.data)\n",
    "print(\"grad_fn:\", y.grad_fn)\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c14dde16-a6be-4151-94cb-96ae98f0648a",
   "metadata": {},
   "source": [
    "Once the operation is executed on a tensor, a new attribute grad_fn is created. This attribute references a function that has created the tensor. In our example, since we squared the tensor, grad_fn will be of type PowBackward0.\n",
    "\n",
    "This grad_fn attribute provides a link to the computational history of the tensor, allowing PyTorch to backpropagate errors and compute gradients when training neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0965e79e-558a-45a9-8ab2-614c503e59c0",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### **Computing Gradients**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36fb6c5b-9b39-4a2f-a767-61032b1b4ffc",
   "metadata": {},
   "source": [
    "Now, let's compute the gradients of `out` with respect to `x`. To do this, we'll call the `backward()` method on the tensor `out`.\n",
    "\n",
    ">[Task]: Compute the gradients of `out` by calling the `backward()` method on it. Afterwards, print the gradients of `x`. The [documentation for backward()](https://pytorch.org/docs/stable/autograd.html#torch.autograd.backward) may be useful.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83685760-bde9-4327-88f7-cfe02bdb3309",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# TODO: Compute the gradient and print it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b1d104b-efef-4fff-869d-8dde1131868e",
   "metadata": {
    "tags": []
   },
   "source": [
    "<details>\n",
    "<summary>Hint (click to reveal)</summary>\n",
    "\n",
    "To compute the gradient:\n",
    "\n",
    "```python\n",
    "y.backward()\n",
    "print(x.grad)\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7f5aecb-8623-481f-a5cf-f8b6dd0c9a37",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### **Gradient Accumulation**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a4df0a1-12a0-4129-a258-915fa8440193",
   "metadata": {},
   "source": [
    "In PyTorch, the gradients of tensors are accumulated into the `.grad` attribute each time you call `.backward()`. This means that if you call `.backward()` multiple times, the gradients will add up.\n",
    "\n",
    "However, by default, calling `.backward()` consumes the computational graph to save memory. If you intend to call `.backward()` multiple times on the same graph, you need to specify `retain_graph=True` during all but the last call.\n",
    "\n",
    ">[Task]: Create a tensor, perform an operation on it, and then call `backward()` twice. Use `retain_graph=True` in the first call to retain the computational graph. Observe the `.grad` attribute after each call.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50a04095-9d7e-48ba-90ed-06718cd379f0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create a tensor\n",
    "w = torch.tensor([1.0], requires_grad=True)\n",
    "\n",
    "# Operation\n",
    "result = w * 2\n",