04-Show-vectors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n",
    "\n",
    "# <!-- TITLE --> [IMDB4] - Reload embedded vectors\n",
    "<!-- DESC --> Retrieving embedded vectors from our trained model\n",
    "<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->\n",
    "\n",
    "## Objectives :\n",
    " - The objective is to retrieve and visualize our embedded vectors\n",
    " - For this, we will use our **previously saved model**.\n",
    "\n",
    "## What we're going to do :\n",
    "\n",
    " - Retrieve our saved model\n",
    " - Extract vectors and play with\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Init python stuff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow.keras as keras\n",
    "import tensorflow.keras.datasets.imdb as imdb\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib\n",
    "import pandas as pd\n",
    "\n",
    "import os,sys,h5py,json,re\n",
    "\n",
    "from importlib import reload\n",
    "\n",
    "sys.path.append('..')\n",
    "import fidle.pwk as pwk\n",
    "\n",
    "run_dir = './run/IMDB2'\n",
    "datasets_dir = pwk.init('IMDB4')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 - Parameters\n",
    "The words in the vocabulary are classified from the most frequent to the rarest.  \n",
    "`vocab_size` is the number of words we will remember in our vocabulary (the other words will be considered as unknown).  \n",
    "`review_len` is the review length  \n",
    "`dictionaries_dir` is where we will go to save our dictionaries. (./data is a good choice)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size           = 10000\n",
    "review_len           = 256\n",
    "\n",
    "dictionaries_dir     = './data'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Override parameters (batch mode) - Just forget this cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwk.override('vocab_size', 'review_len', 'dictionaries_dir')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Get the embedding vectors !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 - Load model and dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(f'{run_dir}/models/best_model.h5')\n",
    "print('Model loaded.')\n",
    "\n",
    "with open(f'{dictionaries_dir}/index_word.json', 'r') as fp:\n",
    "    index_word = json.load(fp)\n",
    "    index_word = { int(i):w for i,w in index_word.items() }\n",
    "    word_index = { w:int(i) for i,w in index_word.items() }\n",
    "    print('Dictionary loaded.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 - Retrieve embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = model.layers[0].get_weights()[0]\n",
    "print('Shape of embeddings : ',embeddings.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 - Build a nice dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_embedding = { index_word[i]:embeddings[i] for i in range(vocab_size) }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Have a look !\n",
    "#### Show embedding of a word :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_embedding['nice']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Few usefull functions to play with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return a l2 distance between 2 words\n",
    "#\n",
    "def l2w(w1,w2):\n",
    "    v1=word_embedding[w1]\n",
    "    v2=word_embedding[w2]\n",
    "    return np.linalg.norm(v2-v1)\n",
    "\n",
    "# Show distance between 2 words \n",
    "#\n",
    "def show_l2(w1,w2):\n",
    "    print(f'\\nL2 between [{w1}] and [{w2}] : ',l2w(w1,w2))\n",
    "\n",
    "# Displays the 15 closest words to a given word\n",
    "#\n",
    "def neighbors(w1):\n",
    "    v1=word_embedding[w1]\n",
    "    dd={}\n",
    "    for i in range(4, 1000):\n",
    "        w2=index_word[i]\n",
    "        dd[w2]=l2w(w1,w2)\n",
    "    dd= {k: v for k, v in sorted(dd.items(), key=lambda item: item[1])}\n",
    "    print(f'\\nNeighbors of [{w1}] : ', list(dd.keys())[1:15])\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_l2('nice', 'pleasant')\n",
    "show_l2('nice', 'horrible')\n",
    "\n",
    "neighbors('horrible')\n",
    "neighbors('great')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwk.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<img width=\"80px\" src=\"../fidle/img/00-Fidle-logo-01.svg\"></img>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}