{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n",
"\n",
"# <!-- TITLE --> [VAE6] - Generation of a clustered dataset\n",
"<!-- DESC --> Episode 2 : Analysis of the CelebA dataset and creation of an clustered and usable dataset\n",
"<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->\n",
"\n",
"## Objectives :\n",
" - Formatting our dataset in **cluster files**, using batch mode\n",
" - Adapting a notebook for batch use\n",
"\n",
"\n",
"The [CelebFaces Attributes Dataset (CelebA)](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) contains about 200,000 images (202599,218,178,3). \n",
"\n",
"\n",
"## What we're going to do :\n",
" - Lire les images\n",
" - redimensionner et normaliser celles-ci,\n",
" - Constituer des clusters d'images en format npy\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1 - Import and init\n",
"### 1.2 - Import"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<style>\n",
"\n",
"div.warn { \n",
" background-color: #fcf2f2;\n",
" border-color: #dFb5b4;\n",
" border-left: 5px solid #dfb5b4;\n",
" padding: 0.5em;\n",
" font-weight: bold;\n",
" font-size: 1.1em;;\n",
" }\n",
"\n",
"\n",
"\n",
"div.nota { \n",
" background-color: #DAFFDE;\n",
" border-left: 5px solid #92CC99;\n",
" padding: 0.5em;\n",
" }\n",
"\n",
"div.todo:before { content:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1My44OTEyIiBoZWlnaHQ9IjE0My4zOTAyIiB2aWV3Qm94PSIwIDAgNTMuODkxMiAxNDMuMzkwMiI+PHRpdGxlPjAwLUJvYi10b2RvPC90aXRsZT48cGF0aCBkPSJNMjMuNDU2OCwxMTQuMzAxNmExLjgwNjMsMS44MDYzLDAsMSwxLDEuODE1NywxLjgyNEExLjgyMDksMS44MjA5LDAsMCwxLDIzLjQ1NjgsMTE0LjMwMTZabS0xMC42NjEyLDEuODIyQTEuODI3MiwxLjgyNzIsMCwxLDAsMTAuOTgsMTE0LjMsMS44MiwxLjgyLDAsMCwwLDEyLjc5NTYsMTE2LjEyMzZabS03LjcwNyw0LjU4NzR2LTVzLjQ4NjMtOS4xMjIzLDguMDIxNS0xMS45Njc1YTE5LjIwODIsMTkuMjA4MiwwLDAsMSw2LjA0ODYtMS4yNDU0LDE5LjE3NzgsMTkuMTc3OCwwLDAsMSw2LjA0ODcsMS4yNDc1YzcuNTM1MSwyLjgzNDcsOC4wMTc0LDExLjk2NzQsOC4wMTc0LDExLjk2NzR2NS4wMjM0bC4wMDQyLDcuNjgydjIuNGMuMDE2Ny4xOTkyLjAzMzYuMzkyMS4wMzM2LjU4NzEsMCwuMjEzOC0uMDE2OC40MTA5LS4wMzM2LjYzMzJ2LjA1ODdoLS4wMDg0YTguMzcxOSw4LjM3MTksMCwwLDEtNy4zNzM4LDcuNjU0N3MtLjk5NTMsMy42MzgtNi42OTMzLDMuNjM4LTYuNjkzNC0zLjYzOC02LjY5MzQtMy42MzhhOC4zNyw4LjM3LDAsMCwxLTcuMzcxNi03LjY1NDdINS4wODQzdi0uMDU4N2MtLjAxODktLjIyLS4wMjk0LS40MTk0LS4wMjk0LS42MzMyLDAtLjE5MjkuMDE2Ny0uMzgzNy4wMjk0LS41ODcxdi0yLjRtMTguMDkzNy00LjA0YTEuMTU2NSwxLjE1NjUsMCwxLDAtMi4zMTI2LDAsMS4xNTY0LDEuMTU2NCwwLDEsMCwyLjMxMjYsMFptNC4wODM0LDBhMS4xNTk1LDEuMTU5NSwwLDEsMC0xLjE2MzYsMS4xN0ExLjE3NSwxLjE3NSwwLDAsMCwyNy4yNjE0LDEyNC4zNzc5Wk05LjM3MzksMTE0LjYzNWMwLDMuMTA5MywyLjQxMzIsMy4zMSwyLjQxMzIsMy4zMWExMzMuOTI0MywxMzMuOTI0MywwLDAsMCwxNC43MzQ4LDBzMi40MTExLS4xOTI5LDIuNDExMS0zLjMxYTguMDc3Myw4LjA3NzMsMCwwLDAtMi40MTExLTUuNTUxOWMtNC41LTMuNTAzMy05LjkxMjYtMy41MDMzLTE0Ljc0MTEsMEE4LjA4NTEsOC4wODUxLDAsMCwwLDkuMzczOSwxMTQuNjM1WiIgc3R5bGU9ImZpbGw6IzAxMDEwMSIvPjxjaXJjbGUgY3g9IjMzLjE0MzYiIGN5PSIxMjQuNTM0IiByPSIzLjgzNjMiIHN0eWxlPSJmaWxsOiMwMTAxMDEiLz48cmVjdCB4PSIzNS42NjU5IiB5PSIxMTIuOTYyNSIgd2lkdGg9IjIuMDc3IiBoZWlnaHQ9IjEwLjU0NTgiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDIxLjYgMjQxLjExMjEpIHJvdGF0ZSgtMTU1Ljc0NikiIHN0eWxlPSJmaWxsOiMwMTAxMDEiLz48Y2lyY2xlIGN4PSIzOC44NzA0IiBjeT0iMTEzLjQyNzkiIHI9IjIuNDA4NSIgc3R5bGU9ImZpbGw6IzAxMDEwMSIvPjxjaXJjbGUgY3g9IjUuMjI0OCIgY3k9IjEyNC41MzQiIHI9IjMuODM2MyIgc3R5bGU9ImZpbGw6IzAxMDEwMSIvPjxyZWN0IHg9IjEuNDE2NCIgeT0iMTI0LjYzMDEiIHdpZHRoPSIyLjA3NyIgaGVpZ2h0PSIxMC41NDU4IiB0cmFuc2Zvcm09InRyYW5zbGF0ZSg0LjkwOTcgMjU5LjgwNikgcm90YXRlKC0xODApIiBzdHlsZT0iZmlsbDojMDEwMTAxIi8+PGNpcmNsZSBjeD0iMi40MDkxIiBjeT0iMTM3LjA5OTYiIHI9IjIuNDA4NSIgc3R5bGU9ImZpbGw6IzAxMDEwMSIvPjxwYXRoIGQ9Ik0xOC4wNTExLDEwMC4xMDY2aC0uMDE0NlYxMDIuNjFoMi4zdi0yLjQyNzlhMi40MjI5LDIuNDIyOSwwLDEsMC0yLjI4NTQtLjA3NTVaIiBzdHlsZT0iZmlsbDojMDEwMTAxIi8+PHBhdGggZD0iTTM5LjQyMTQsMjcuMjU4djEuMDVBMTEuOTQ1MiwxMS45NDUyLDAsMCwwLDQ0LjU5NTQsNS43OWEuMjQ0OS4yNDQ5LDAsMCwxLS4wMjM1LS40MjI3TDQ2Ljc1LDMuOTUxNWEuMzg5Mi4zODkyLDAsMCwxLC40MjYyLDAsMTQuODQ0MiwxNC44NDQyLDAsMCwxLTcuNzU0MywyNy4yNTkxdjEuMDY3YS40NS40NSwwLDAsMS0uNzA0Ny4zNzU4bC0zLjg0MTktMi41MWEuNDUuNDUsMCwwLDEsMC0uNzUxNmwzLjg0MTktMi41MWEuNDUuNDUsMCwwLDEsLjY5NDYuMzc1OFpNNDMuMjMsMi41ODkyLDM5LjM4NzguMDc5NGEuNDUuNDUsMCwwLDAtLjcwNDYuMzc1OHYxLjA2N2ExNC44NDQyLDE0Ljg0NDIsMCwwLDAtNy43NTQzLDI3LjI1OTEuMzg5LjM4OSwwLDAsMCwuNDI2MSwwbDIuMTc3Ny0xLjQxOTNhLjI0NS4yNDUsMCwwLDAtLjAyMzUtLjQyMjgsMTEuOTQ1MSwxMS45NDUxLDAsMCwxLDUuMTc0LTIyLjUxNDZ2MS4wNWEuNDUuNDUsMCwwLDAsLjcwNDYuMzc1OGwzLjg1NTMtMi41MWEuNDUuNDUsMCwwLDAsMC0uNzUxNlpNMzkuMDUyMywxNC4yNDU4YTIuMTIwNiwyLjEyMDYsMCwxLDAsMi4xMjA2LDIuMTIwNmgwQTIuMTI0LDIuMTI0LDAsMCwwLDM5LjA1MjMsMTQuMjQ1OFptNi4wNzMyLTQuNzc4MS44MjU0LjgyNTVhMS4wNTY4LDEuMDU2OCwwLDAsMSwuMTE3NSwxLjM0MjFsLS44MDIsMS4xNDQyYTcuMTAxOCw3LjEwMTgsMCwwLDEsLjcxMTQsMS43MTEybDEuMzc1Ny4yNDE2YTEuMDU2OSwxLjA1NjksMCwwLDEsLjg3NTcsMS4wNHYxLjE2NDNhMS4wNTY5LDEuMDU2OSwwLDAsMS0uODc1NywxLjA0bC0xLjM3MjQuMjQxNkE3LjExLDcuMTEsMCwwLDEsNDUuMjcsMTkuOTNsLjgwMTksMS4xNDQyYTEuMDU3LDEuMDU3LDAsMCwxLS4xMTc0LDEuMzQyMmwtLjgyODguODQ4OWExLjA1NywxLjA1NywwLDAsMS0xLjM0MjEuMTE3NGwtMS4xNDQyLS44MDE5YTcuMTMzOCw3LjEzMzgsMCwwLDEtMS43MTEzLjcxMTNsLS4yNDE2LDEuMzcyNGExLjA1NjgsMS4wNTY4LDAsMCwxLTEuMDQuODc1N0gzOC40Njg0YTEuMDU2OCwxLjA1NjgsMCwwLDEtMS4wNC0uODc1N2wtLjI0MTYtMS4zNzI0YTcuMTM1NSw3LjEzNTUsMCwwLDEtMS43MTEzLS43MTEzbC0xLjE0NDEuODAxOWExLjA1NzEsMS4wNTcxLDAsMCwxLTEuMzQyMi0uMTE3NGwtLjgzNTUtLjgyNTVhMS4wNTcsMS4wNTcsMCwwLDEtLjExNzQtMS4zNDIxbC44MDE5LTEuMTQ0MmE3LjEyMSw3LjEyMSwwLDAsMS0uNzExMy0xLjcxMTJsLTEuMzcyNC0uMjQxNmExLjA1NjksMS4wNTY5LDAsMCwxLS44NzU3LTEuMDRWMTUuNzgyNmExLjA1NjksMS4wNTY5LDAsMCwxLC44NzU3LTEuMDRsMS4zNzU3LS4yNDE2YTcuMTEsNy4xMSwwLDAsMSwuNzExNC0xLjcxMTJsLS44MDItMS4xNDQyYTEuMDU3LDEuMDU3LDAsMCwxLC4xMTc1LTEuMzQyMmwuODI1NC0uODI1NEExLjA1NjgsMS4wNTY4LDAsMCwxLDM0LjMyNDUsOS4zNmwxLjE0NDIuODAxOUE3LjEzNTUsNy4xMzU1LDAsMCwxLDM3LjE4LDkuNDUxbC4yNDE2LTEuMzcyNGExLjA1NjgsMS4wNTY4LDAsMCwxLDEuMDQtLjg3NTdoMS4xNjc3YTEuMDU2OSwxLjA1NjksMCwwLDEsMS4wNC44NzU3bC4yNDE2LDEuMzcyNGE3LjEyNSw3LjEyNSwwLDAsMSwxLjcxMTIuNzExM0w0My43NjY2LDkuMzZBMS4wNTY5LDEuMDU2OSwwLDAsMSw0NS4xMjU1LDkuNDY3N1ptLTIuMDMsNi44OTg3QTQuMDQzMyw0LjA0MzMsMCwxLDAsMzkuMDUyMywyMC40MWgwQTQuMDQ2NSw0LjA0NjUsMCwwLDAsNDMuMDk1NSwxNi4zNjY0WiIgc3R5bGU9ImZpbGw6I2UxMjIyOSIvPjxwb2x5Z29uIHBvaW50cz0iMzkuNDEzIDM0Ljc1NyAzOS41MzcgMzQuNzU3IDM5LjY3NSAzNC43NTcgMzkuNjc1IDEwOS41MSAzOS41MzcgMTA5LjUxIDM5LjQxMyAxMDkuNTEgMzkuNDEzIDM0Ljc1NyAzOS40MTMgMzQuNzU3IiBzdHlsZT0iZmlsbDpub25lO3N0cm9rZTojOTk5O3N0cm9rZS1saW5lY2FwOnJvdW5kO3N0cm9rZS1taXRlcmxpbWl0OjEwO3N0cm9rZS13aWR0aDowLjMwODg1NDQ1MDU2MDE2MThweDtmaWxsLXJ1bGU6ZXZlbm9kZCIvPjwvc3ZnPg==);\n",
" float:left;\n",
" margin-right:20px;\n",
" margin-top:-20px;\n",
" margin-bottom:20px;\n",
"}\n",
"div.todo{\n",
" font-weight: bold;\n",
" font-size: 1.1em;\n",
" margin-top:40px;\n",
"}\n",
"div.todo ul{\n",
" margin: 0.2em;\n",
"}\n",
"div.todo li{\n",
" margin-left:60px;\n",
" margin-top:0;\n",
" margin-bottom:0;\n",
"}\n",
"\n",
"div .comment{\n",
" font-size:0.8em;\n",
" color:#696969;\n",
"}\n",
"\n",
"\n",
"\n",
"</style>\n",
"\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"<br>**FIDLE 2020 - Practical Work Module**"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Version : 2.0.7\n",
"Notebook id : VAE6\n",
"Run time : Wednesday 27 January 2021, 09:48:49\n",
"TensorFlow version : 2.2.0\n",
"Keras version : 2.3.0-tf\n",
"Datasets dir : /gpfswork/rech/mlh/uja62cb/datasets\n",
"Run dir : ./run\n",
"Update keras cache : False\n"
]
}
],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"from skimage import io, transform\n",
"\n",
"import os,pathlib,time,sys,json,glob\n",
"import csv\n",
"import math, random\n",
"\n",
"from importlib import reload\n",
"\n",
"sys.path.append('..')\n",
"import fidle.pwk as pwk\n",
"\n",
"datasets_dir = pwk.init('VAE6')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2 Parameters\n",
"All the dataset will be use for training \n",
"Reading the 200,000 images can take a long time **(>20 minutes)** and a lot of place **(>170 GB)** \n",
"Example : \n",
"Image Sizes: 128x128 : 74 GB \n",
"Image Sizes: 192x160 : 138 GB \n",
"\n",
"You can define theses parameters : \n",
"`scale` : 1 mean 100% of the dataset - set 0.05 for tests \n",
"`image_size` : images size in the clusters, should be 128x128 or 192,160 (original is 218,178) \n",
"`output_dir` : where to write clusters, could be :\n",
" - `./data`, for tests purpose\n",
" - `<datasets_dir>/celeba/enhanced` to add clusters in your datasets dir. \n",
" \n",
"`cluster_size` : number of images in a cluster, 10000 is fine. (will be adjust by scale)\n",
"\n",
"**Note :** If the target folder is not empty and exit_if_exist is True, the construction is blocked. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# ---- Parameters you can change -----------------------------------\n",
"\n",
"# ---- Tests\n",
"scale = 0.02\n",
"cluster_size = 10000\n",
"image_size = (128,128)\n",
"output_dir = './data'\n",
"exit_if_exist = False\n",
"\n",
"# ---- Full clusters generation, medium size\n",
"# scale = 1.\n",
"# cluster_size = 10000\n",
"# image_size = (128,128)\n",
"# output_dir = f'{datasets_dir}/celeba/enhanced'\n",
"# exit_if_exist = True\n",
"\n",
"# ---- Full clusters generation, large size\n",
"# scale = 1.\n",
"# cluster_size = 10000\n",
"# image_size = (192,160)\n",
"# output_dir = f'{datasets_dir}/celeba/enhanced'\n",
"# exit_if_exist = True"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# ---- Used for continous integration - Just forget this line\n",
"#\n",
"pwk.override('scale', 'cluster_size', 'image_size', 'output_dir', 'exit_if_exist')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2 - Directories and files :"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"dataset_csv = f'{datasets_dir}/celeba/origine/list_attr_celeba.csv'\n",
"dataset_img = f'{datasets_dir}/celeba/origine/img_align_celeba'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2 - Read and shuffle filenames catalog"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"dataset_desc = pd.read_csv(dataset_csv, header=0)\n",
"dataset_desc = dataset_desc.reindex(np.random.permutation(dataset_desc.index))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3 - Save as clusters of n images"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.2 - Cooking function"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def read_and_save( dataset_img, dataset_desc, scale=1,\n",
" cluster_size=1000, cluster_dir='./dataset_cluster', cluster_name='images',\n",
" image_size=(128,128),\n",
" exit_if_exist=True):\n",
" global pwk\n",
" \n",
" def save_cluster(imgs,desc,cols,id):\n",
" file_img = f'{cluster_dir}/{cluster_name}-{id:03d}.npy'\n",
" file_desc = f'{cluster_dir}/{cluster_name}-{id:03d}.csv'\n",
" np.save(file_img, np.array(imgs))\n",
" df=pd.DataFrame(data=desc,columns=cols)\n",
" df.to_csv(file_desc, index=False)\n",
" return [],[],id+1\n",
" \n",
" pwk.chrono_start()\n",
" cols = list(dataset_desc.columns)\n",
"\n",
" # ---- Check if cluster files exist\n",
" #\n",
" if exit_if_exist and os.path.isfile(f'{cluster_dir}/images-000.npy'):\n",
" print('\\n*** Oups. There are already clusters in the target folder!\\n')\n",
" return 0,0\n",
" pwk.mkdir(cluster_dir)\n",
"\n",
" # ---- Scale\n",
" #\n",
" n=int(len(dataset_desc)*scale)\n",
" dataset = dataset_desc[:n]\n",
" cluster_size = int(cluster_size*scale)\n",
" pwk.subtitle('Parameters :')\n",
" print(f'Scale is : {scale}')\n",
" print(f'Image size is : {image_size}')\n",
" print(f'dataset length is : {n}')\n",
" print(f'cluster size is : {cluster_size}')\n",
" print(f'clusters nb is :',int(n/cluster_size + 1))\n",
" print(f'cluster dir is : {cluster_dir}')\n",
" \n",
" # ---- Read and save clusters\n",
" #\n",
" pwk.subtitle('Running...')\n",
" imgs, desc, cluster_id = [],[],0\n",
" #\n",
" for i,row in dataset.iterrows():\n",
" #\n",
" filename = f'{dataset_img}/{row.image_id}'\n",
" #\n",
" # ---- Read image, resize (and normalize)\n",
" #\n",
" img = io.imread(filename)\n",
" img = transform.resize(img, image_size)\n",
" #\n",
" # ---- Add image and description\n",
" #\n",
" imgs.append( img )\n",
" desc.append( row.values )\n",
" #\n",
" # ---- Progress bar\n",
" #\n",
" pwk.update_progress(f'Cluster {cluster_id:03d} :',len(imgs),cluster_size)\n",
" #\n",
" # ---- Save cluster if full\n",
" #\n",
" if len(imgs)==cluster_size:\n",
" imgs,desc,cluster_id=save_cluster(imgs,desc,cols, cluster_id)\n",
"\n",
" # ---- Save uncomplete cluster\n",
" if len(imgs)>0 : imgs,desc,cluster_id=save_cluster(imgs,desc,cols,cluster_id)\n",
"\n",
" duration=pwk.chrono_stop()\n",
" return cluster_id,duration\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.3 - Cluster building"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/markdown": [
"<br>**Parameters :**"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Scale is : 0.02\n",
"Image size is : (128, 128)\n",
"dataset length is : 4051\n",
"cluster size is : 200\n",
"clusters nb is : 21\n",
"cluster dir is : ./data/clusters-128x128\n"
]
},
{
"data": {
"text/markdown": [
"<br>**Running...**"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cluster 000 : [########################################] 100.0% of 200\n",
"Cluster 001 : [########################################] 100.0% of 200\n",
"Cluster 002 : [########################################] 100.0% of 200\n",
"Cluster 003 : [########################################] 100.0% of 200\n",
"Cluster 004 : [########################################] 100.0% of 200\n",
"Cluster 005 : [########################################] 100.0% of 200\n",
"Cluster 006 : [########################################] 100.0% of 200\n",
"Cluster 007 : [########################################] 100.0% of 200\n",
"Cluster 008 : [########################################] 100.0% of 200\n",
"Cluster 009 : [########################################] 100.0% of 200\n",
"Cluster 010 : [########################################] 100.0% of 200\n",
"Cluster 011 : [########################################] 100.0% of 200\n",
"Cluster 012 : [########################################] 100.0% of 200\n",
"Cluster 013 : [########################################] 100.0% of 200\n",
"Cluster 014 : [########################################] 100.0% of 200\n",
"Cluster 015 : [########################################] 100.0% of 200\n",
"Cluster 016 : [########################################] 100.0% of 200\n",
"Cluster 017 : [########################################] 100.0% of 200\n",
"Cluster 018 : [########################################] 100.0% of 200\n",
"Cluster 019 : [########################################] 100.0% of 200\n",
"Cluster 020 : [##########------------------------------] 25.0% of 200\r"
]
},
{
"data": {
"text/markdown": [
"<br>**Conclusion :**"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Duration : 0:01:57\n",
"Size : 1.5 Go\n"
]
}
],
"source": [
"# ---- Build clusters\n",
"#\n",
"lx,ly = image_size\n",
"cluster_dir = f'{output_dir}/clusters-{lx}x{ly}'\n",
"\n",
"cluster_nb,duration = read_and_save( dataset_img, dataset_desc,\n",
" scale = scale,\n",
" cluster_size = cluster_size, \n",
" cluster_dir = cluster_dir,\n",
" image_size = image_size,\n",
" exit_if_exist = exit_if_exist)\n",
"\n",
"# ---- Conclusion...\n",
"\n",
"directory = pathlib.Path(cluster_dir)\n",
"s=sum(f.stat().st_size for f in directory.glob('**/*') if f.is_file())\n",
"\n",
"pwk.subtitle('Conclusion :')\n",
"print('Duration : ',pwk.hdelay(duration))\n",
"print('Size : ',pwk.hsize(s))"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"End time is : Wednesday 27 January 2021, 09:50:47\n",
"Duration is : 00:01:58 822ms\n",
"This notebook ends here\n"
]
}
],
"source": [
"pwk.end()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"<img width=\"80px\" src=\"../fidle/img/00-Fidle-logo-01.svg\"></img>"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}