diff --git a/fidle/01 - Update index.ipynb b/fidle/01-update-index.ipynb similarity index 100% rename from fidle/01 - Update index.ipynb rename to fidle/01-update-index.ipynb diff --git a/fidle/02 - Running test.ipynb b/fidle/02 - Running test.ipynb deleted file mode 100644 index db2326d8bcb399f237311a3602ece9d7ddaa50b2..0000000000000000000000000000000000000000 --- a/fidle/02 - Running test.ipynb +++ /dev/null @@ -1,168 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n", - "\n", - "\n", - "# Gestion des tests d'intégration continue\n", - "\n", - "**La liste des notebooks a éxécuter** et de leurs paramètres (override) est définie dans un **profile**. \n", - " - Un profile est un fichier yaml\n", - " - Un profile par défaut est généré en (1).\n", - "\n", - "Lorqu'un **profile** est prêt, il peut être soumis pour **éxécution** (2)\n", - " - Pour chaque notebook, les paramètres sont positionnés (variables d'env.)\n", - " - Le notebook est lu, puis éxécuté\n", - " - Après éxécution, le notebook est sauvegardé sous son nom moodifié (output_tag)\n", - "\n", - "Un **rapport d'éxécution** est généré durant l'éxécution des tests : [./logs/ci_report.json](./logs/ci_report.json) \n", - "A la fin des tests, ce rapport peut être visualisé via le notebook : [./03-ci_report.ipynb](./03%20-%20ci_report.ipynb)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import cookci" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Get and Save default profile\n", - "Génère un profile par défaut comprenant tous les notebooks du moment... \n", - "...en tout cas présent dans catalog.json ;-)" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Catalog saved as ./ci/default.yml\n", - "Entries : 36\n" - ] - } - ], - "source": [ - "profile = cookci.get_ci_profile()\n", - "cookci.save_profile(profile, './ci/default.yml')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Load a saved profile" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "./ci/smart.yml loaded.\n", - "Entries : 3\n" - ] - } - ], - "source": [ - "smart_profile = cookci.load_profile('./ci/smart.yml')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Run this profile" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Finished file has been reset.\n", - "\n", - "\n", - "Notebook : LINR1\n", - " Run notebook.....done.\n", - " Duration : 0:00:04\n", - " Saved as : 01-Linear-Regression==done==.ipynb\n", - "\n", - "Notebook : PER57\n", - " Run notebook.....done.\n", - " Duration : 0:00:02\n", - " Saved as : 01-Simple-Perceptron==done==.ipynb\n", - "\n", - "Notebook : GTSRB2\n", - " set overrides :\n", - " FIDLE_OVERRIDE_GTSRB2_run_dir = ./run/GTSRB2_ci\n", - " FIDLE_OVERRIDE_GTSRB2_dataset_name = set-24x24-L\n", - " FIDLE_OVERRIDE_GTSRB2_batch_size = 32\n", - " FIDLE_OVERRIDE_GTSRB2_epochs = 3\n", - " FIDLE_OVERRIDE_GTSRB2_scale = 0.1\n", - " Run notebook...\n", - " ************************************************************\n", - " ** AAARG.. An error occured : CellExecutionError\n", - " ** See notebook : 02-First-convolutions==ERROR==.ipynb for details.\n", - " ************************************************************\n", - " Duration : 0:00:02\n", - " Saved as : 02-First-convolutions==ERROR==.ipynb\n" - ] - } - ], - "source": [ - "cookci.reset_ci_report()\n", - "cookci.run_profile(smart_profile)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "<img width=\"80px\" src=\"../fidle/img/00-Fidle-logo-01.svg\"></img>" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.9" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/fidle/02-running-ci-tests.ipynb b/fidle/02-running-ci-tests.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..13cd7325f4ced47667ee18c20b87930d11805bf6 --- /dev/null +++ b/fidle/02-running-ci-tests.ipynb @@ -0,0 +1,235 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n", + "\n", + "\n", + "# Gestion des tests d'intégration continue\n", + "\n", + "**La liste des notebooks a éxécuter** et de leurs paramètres (override) est définie dans un **profile**. \n", + " - Un profile est un fichier yaml\n", + " - Un profile par défaut (à complété !) peut être généré.\n", + "\n", + "Lorqu'un **profile** est prêt, il peut être soumis pour **éxécution**\n", + " - Pour chaque notebook, les paramètres sont positionnés (variables d'env.)\n", + " - Le notebook est lu, puis éxécuté\n", + " - Après éxécution, le notebook est sauvegardé sous son nom taggé (output_tag)\n", + "\n", + "Un **rapport d'éxécution** est généré durant l'éxécution des tests. \n", + "A la fin des tests, ce rapport peut être visualisé via le notebook : [./03-ci-report.ipynb](./03-ci-report.ipynb)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import cookci" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Save a default profile\n", + "Génère un profile par défaut comprenant tous les notebooks du moment... \n", + "...en tout cas présent dans catalog.json ;-)" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "# profile = cookci.get_ci_profile()\n", + "# cookci.save_profile(profile, './ci/default.yml')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Run a profile" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Run profile session - FIDLE 2021\n", + "Version : 1.0\n", + "\n", + "Load profile :./ci/smart_cpu.yml\n", + " Entries : 15\n", + "\n", + "Create new ci report : /home/pjluc/dev/fidle/fidle/logs/ci_report_smart.json\n", + "\n", + "Notebook : LINR1\n", + " Run notebook.....done.\n", + " Duration : 0:00:03\n", + " Saved as : 01-Linear-Regression==done==.ipynb\n", + "\n", + "Notebook : GRAD1\n", + " Run notebook.....done.\n", + " Duration : 0:00:06\n", + " Saved as : 02-Gradient-descent==done==.ipynb\n", + "\n", + "Notebook : POLR1\n", + " Run notebook.....done.\n", + " Duration : 0:00:03\n", + " Saved as : 03-Polynomial-Regression==done==.ipynb\n", + "\n", + "Notebook : LOGR1\n", + " Run notebook.....done.\n", + " Duration : 0:00:03\n", + " Saved as : 04-Logistic-Regression==done==.ipynb\n", + "\n", + "Notebook : PER57\n", + " Run notebook.....done.\n", + " Duration : 0:00:02\n", + " Saved as : 01-Simple-Perceptron==done==.ipynb\n", + "\n", + "Notebook : BHPD1\n", + " Run notebook.....done.\n", + " Duration : 0:00:13\n", + " Saved as : 01-DNN-Regression==done==.ipynb\n", + "\n", + "Notebook : BHPD2\n", + " Run notebook.....done.\n", + " Duration : 0:00:13\n", + " Saved as : 02-DNN-Regression-Premium==done==.ipynb\n", + "\n", + "Notebook : MNIST1\n", + " Run notebook.....done.\n", + " Duration : 0:00:37\n", + " Saved as : 01-DNN-MNIST==done==.ipynb\n", + "\n", + "Notebook : GTSRB1\n", + " set overrides :\n", + " FIDLE_OVERRIDE_GTSRB1_scale = 0.05\n", + " FIDLE_OVERRIDE_GTSRB1_output_dir = ./data\n", + " Run notebook.....done.\n", + " Duration : 0:01:24\n", + " Saved as : 01-Preparation-of-data==done==.ipynb\n", + "\n", + "Notebook : GTSRB2\n", + " set overrides :\n", + " FIDLE_OVERRIDE_GTSRB2_run_dir = ./run/GTSRB2_ci\n", + " FIDLE_OVERRIDE_GTSRB2_dataset_name = set-24x24-L\n", + " FIDLE_OVERRIDE_GTSRB2_batch_size = 64\n", + " FIDLE_OVERRIDE_GTSRB2_epochs = 5\n", + " FIDLE_OVERRIDE_GTSRB2_scale = 1\n", + " Run notebook.....done.\n", + " Duration : 0:03:55\n", + " Saved as : 02-First-convolutions==done==.ipynb\n", + "\n", + "Notebook : GTSRB3\n", + " set overrides :\n", + " FIDLE_OVERRIDE_GTSRB3_run_dir = ./run/GTSRB3_ci\n", + " FIDLE_OVERRIDE_GTSRB3_dataset_name = set-24x24-L\n", + " FIDLE_OVERRIDE_GTSRB3_batch_size = 64\n", + " FIDLE_OVERRIDE_GTSRB3_epochs = 5\n", + " FIDLE_OVERRIDE_GTSRB3_scale = 1\n", + " Run notebook.....done.\n", + " Duration : 0:04:22\n", + " Saved as : 03-Tracking-and-visualizing==done==.ipynb\n", + "\n", + "Notebook : GTSRB4\n", + " set overrides :\n", + " FIDLE_OVERRIDE_GTSRB4_run_dir = ./run/GTSRB4_ci\n", + " FIDLE_OVERRIDE_GTSRB4_dataset_name = set-24x24-L\n", + " FIDLE_OVERRIDE_GTSRB4_batch_size = 64\n", + " FIDLE_OVERRIDE_GTSRB4_epochs = 5\n", + " FIDLE_OVERRIDE_GTSRB4_scale = 0.5\n", + " Run notebook...\n", + " ************************************************************\n", + " ** AAARG.. An error occured : AssertionError\n", + " ** See notebook : 04-Data-augmentation==ERROR==.ipynb for details.\n", + " ************************************************************\n", + " Duration : 0:02:12\n", + " Saved as : 04-Data-augmentation==ERROR==.ipynb\n", + "\n", + "Notebook : GTSRB5\n", + " set overrides :\n", + " FIDLE_OVERRIDE_GTSRB5_run_dir = ./run/GTSRB5_ci\n", + " FIDLE_OVERRIDE_GTSRB5_datasets = ['set-24x24-L', 'set-24x24-RGB', 'set-48x48-RGB']\n", + " FIDLE_OVERRIDE_GTSRB5_models = {'v1':'get_model_v1', 'v2':'get_model_v2', 'v3':'get_model_v3'}\n", + " FIDLE_OVERRIDE_GTSRB5_batch_size = 64\n", + " FIDLE_OVERRIDE_GTSRB5_epochs = 5\n", + " FIDLE_OVERRIDE_GTSRB5_scale = 0.1\n", + " FIDLE_OVERRIDE_GTSRB5_with_datagen = True\n", + " FIDLE_OVERRIDE_GTSRB5_verbose = 0\n", + " Run notebook...\n", + " ************************************************************\n", + " ** AAARG.. An error occured : CellExecutionError\n", + " ** See notebook : 05-Full-convolutions==ERROR==.ipynb for details.\n", + " ************************************************************\n", + " Duration : 0:00:02\n", + " Saved as : 05-Full-convolutions==ERROR==.ipynb\n", + "\n", + "Notebook : GTSRB6\n", + " Run notebook.....done.\n", + " Duration : 0:00:03\n", + " Saved as : 06-Notebook-as-a-batch==done==.ipynb\n", + "\n", + "Notebook : GTSRB7\n", + " set overrides :\n", + " FIDLE_OVERRIDE_GTSRB7_run_dir = ./run/GTSRB7_ci\n", + " FIDLE_OVERRIDE_GTSRB7_report_dir = ./run/GTSRB5\n", + " Run notebook.....done.\n", + " Duration : 0:00:02\n", + " Saved as : 07-Show-report==done==.ipynb\n", + "\n", + "End of running process\n", + " Duration : 0:13:29\n", + "\n", + "Complete ci report : /home/pjluc/dev/fidle/fidle/logs/ci_report_smart.json\n" + ] + } + ], + "source": [ + "cookci.run_profile('./ci/smart_cpu.yml', report_name='./logs/ci_report_smart.json')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "<img width=\"80px\" src=\"../fidle/img/00-Fidle-logo-01.svg\"></img>" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.9" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/fidle/03 - ci_report.ipynb b/fidle/03 - ci_report.ipynb deleted file mode 100644 index 1bb74c814e624d3d46d5b6d0e63e24589b77414c..0000000000000000000000000000000000000000 --- a/fidle/03 - ci_report.ipynb +++ /dev/null @@ -1,116 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n", - "\n", - "\n", - "## Continuous integration report\n", - "\n", - "Each notebook indicates its start and end of execution:\n", - " - at the beginning, with `pwk.init()`\n", - " - at the end, with `pwk.end()` \n", - " \n", - "All of theses informations are saved in a json file - Final report is below and in a HTML file. \n", - "See ./log" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "<style type=\"text/css\" >\n", - " #T_58cf4_ td {\n", - " font-size: 110%;\n", - " text-align: left;\n", - " } #T_58cf4_ th {\n", - " font-size: 110%;\n", - " text-align: left;\n", - " }#T_58cf4_row2_col7{\n", - " background-color: OrangeRed;\n", - " color: white;\n", - " background-color: OrangeRed;\n", - " color: white;\n", - " }</style><table id=\"T_58cf4_\" ><thead> <tr> <th class=\"col_heading level0 col0\" >id</th> <th class=\"col_heading level0 col1\" >dir</th> <th class=\"col_heading level0 col2\" >src</th> <th class=\"col_heading level0 col3\" >out</th> <th class=\"col_heading level0 col4\" >start</th> <th class=\"col_heading level0 col5\" >end</th> <th class=\"col_heading level0 col6\" >duration</th> <th class=\"col_heading level0 col7\" >state</th> </tr></thead><tbody>\n", - " <tr>\n", - " <td id=\"T_58cf4_row0_col0\" class=\"data row0 col0\" ><a href='../LinearReg/01-Linear-Regression.ipynb'>LINR1</a></td>\n", - " <td id=\"T_58cf4_row0_col1\" class=\"data row0 col1\" >LinearReg</td>\n", - " <td id=\"T_58cf4_row0_col2\" class=\"data row0 col2\" ><a href='../LinearReg/01-Linear-Regression.ipynb'>01-Linear-Regression.ipynb</a></td>\n", - " <td id=\"T_58cf4_row0_col3\" class=\"data row0 col3\" ><a href='../LinearReg/01-Linear-Regression==done==.ipynb'>01-Linear-Regression==done==.ipynb</a></td>\n", - " <td id=\"T_58cf4_row0_col4\" class=\"data row0 col4\" >11/01/21 01:07:35</td>\n", - " <td id=\"T_58cf4_row0_col5\" class=\"data row0 col5\" >11/01/21 01:07:39</td>\n", - " <td id=\"T_58cf4_row0_col6\" class=\"data row0 col6\" >0:00:04</td>\n", - " <td id=\"T_58cf4_row0_col7\" class=\"data row0 col7\" >ok</td>\n", - " </tr>\n", - " <tr>\n", - " <td id=\"T_58cf4_row1_col0\" class=\"data row1 col0\" ><a href='../IRIS/01-Simple-Perceptron.ipynb'>PER57</a></td>\n", - " <td id=\"T_58cf4_row1_col1\" class=\"data row1 col1\" >IRIS</td>\n", - " <td id=\"T_58cf4_row1_col2\" class=\"data row1 col2\" ><a href='../IRIS/01-Simple-Perceptron.ipynb'>01-Simple-Perceptron.ipynb</a></td>\n", - " <td id=\"T_58cf4_row1_col3\" class=\"data row1 col3\" ><a href='../IRIS/01-Simple-Perceptron==done==.ipynb'>01-Simple-Perceptron==done==.ipynb</a></td>\n", - " <td id=\"T_58cf4_row1_col4\" class=\"data row1 col4\" >11/01/21 01:07:39</td>\n", - " <td id=\"T_58cf4_row1_col5\" class=\"data row1 col5\" >11/01/21 01:07:42</td>\n", - " <td id=\"T_58cf4_row1_col6\" class=\"data row1 col6\" >0:00:02</td>\n", - " <td id=\"T_58cf4_row1_col7\" class=\"data row1 col7\" >ok</td>\n", - " </tr>\n", - " <tr>\n", - " <td id=\"T_58cf4_row2_col0\" class=\"data row2 col0\" ><a href='../GTSRB/02-First-convolutions.ipynb'>GTSRB2</a></td>\n", - " <td id=\"T_58cf4_row2_col1\" class=\"data row2 col1\" >GTSRB</td>\n", - " <td id=\"T_58cf4_row2_col2\" class=\"data row2 col2\" ><a href='../GTSRB/02-First-convolutions.ipynb'>02-First-convolutions.ipynb</a></td>\n", - " <td id=\"T_58cf4_row2_col3\" class=\"data row2 col3\" ><a href='../GTSRB/02-First-convolutions==ERROR==.ipynb'>02-First-convolutions==ERROR==.ipynb</a></td>\n", - " <td id=\"T_58cf4_row2_col4\" class=\"data row2 col4\" >11/01/21 01:07:42</td>\n", - " <td id=\"T_58cf4_row2_col5\" class=\"data row2 col5\" >11/01/21 01:07:44</td>\n", - " <td id=\"T_58cf4_row2_col6\" class=\"data row2 col6\" >0:00:02</td>\n", - " <td id=\"T_58cf4_row2_col7\" class=\"data row2 col7\" >ERROR</td>\n", - " </tr>\n", - " </tbody></table>" - ], - "text/plain": [ - "<pandas.io.formats.style.Styler at 0x7f1cf038af10>" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "import cookci\n", - "cookci.build_ci_report()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "<img width=\"80px\" src=\"../fidle/img/00-Fidle-logo-01.svg\"></img>" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.9" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/fidle/03-ci-report.ipynb b/fidle/03-ci-report.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..1ad3362b15a960c0ea1e0561c8b4f370e15d8d3d --- /dev/null +++ b/fidle/03-ci-report.ipynb @@ -0,0 +1,153 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n", + "\n", + "\n", + "## Continuous integration report\n", + "\n", + "Réalisation des tests : [02-running-ci-tests.ipynb](02-running-ci-tests.ipynb) \n", + "Résultats bruts : [./logs/ci_report.json](./logs/ci_report.json) " + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "### About :" + ], + "text/plain": [ + "<IPython.core.display.Markdown object>" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/markdown": [ + "**Version** : 1.0 \n", + "**Output_Tag** : ==done== \n", + "**Save_Figs** : False \n", + "**Description** : A basic profile example \n", + "**Host** : Oban \n", + "**Profile** : ./ci/smart.yml \n", + "**Start** : 11/01/21 16:20:47 \n", + "**End** : 11/01/21 16:20:52 \n", + "**Duration** : 0:00:05 \n" + ], + "text/plain": [ + "<IPython.core.display.Markdown object>" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/markdown": [ + "### Details :" + ], + "text/plain": [ + "<IPython.core.display.Markdown object>" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "<style type=\"text/css\" >\n", + " #T_5a52f_ td {\n", + " font-size: 110%;\n", + " text-align: left;\n", + " } #T_5a52f_ th {\n", + " font-size: 110%;\n", + " text-align: left;\n", + " }</style><table id=\"T_5a52f_\" ><thead> <tr> <th class=\"col_heading level0 col0\" >Id</th> <th class=\"col_heading level0 col1\" >Dir</th> <th class=\"col_heading level0 col2\" >Src</th> <th class=\"col_heading level0 col3\" >Out</th> <th class=\"col_heading level0 col4\" >Start</th> <th class=\"col_heading level0 col5\" >End</th> <th class=\"col_heading level0 col6\" >Duration</th> <th class=\"col_heading level0 col7\" >State</th> </tr></thead><tbody>\n", + " <tr>\n", + " <td id=\"T_5a52f_row0_col0\" class=\"data row0 col0\" ><a href='../LinearReg/01-Linear-Regression.ipynb'>LINR1</a></td>\n", + " <td id=\"T_5a52f_row0_col1\" class=\"data row0 col1\" >LinearReg</td>\n", + " <td id=\"T_5a52f_row0_col2\" class=\"data row0 col2\" ><a href='../LinearReg/01-Linear-Regression.ipynb'>01-Linear-Regression.ipynb</a></td>\n", + " <td id=\"T_5a52f_row0_col3\" class=\"data row0 col3\" ><a href='../LinearReg/01-Linear-Regression==done==.ipynb'>01-Linear-Regression==done==.ipynb</a></td>\n", + " <td id=\"T_5a52f_row0_col4\" class=\"data row0 col4\" >11/01/21 16:20:47</td>\n", + " <td id=\"T_5a52f_row0_col5\" class=\"data row0 col5\" >11/01/21 16:20:50</td>\n", + " <td id=\"T_5a52f_row0_col6\" class=\"data row0 col6\" >0:00:02</td>\n", + " <td id=\"T_5a52f_row0_col7\" class=\"data row0 col7\" >ok</td>\n", + " </tr>\n", + " <tr>\n", + " <td id=\"T_5a52f_row1_col0\" class=\"data row1 col0\" ><a href='../IRIS/01-Simple-Perceptron.ipynb'>PER57</a></td>\n", + " <td id=\"T_5a52f_row1_col1\" class=\"data row1 col1\" >IRIS</td>\n", + " <td id=\"T_5a52f_row1_col2\" class=\"data row1 col2\" ><a href='../IRIS/01-Simple-Perceptron.ipynb'>01-Simple-Perceptron.ipynb</a></td>\n", + " <td id=\"T_5a52f_row1_col3\" class=\"data row1 col3\" ><a href='../IRIS/01-Simple-Perceptron==done==.ipynb'>01-Simple-Perceptron==done==.ipynb</a></td>\n", + " <td id=\"T_5a52f_row1_col4\" class=\"data row1 col4\" >11/01/21 16:20:50</td>\n", + " <td id=\"T_5a52f_row1_col5\" class=\"data row1 col5\" >11/01/21 16:20:52</td>\n", + " <td id=\"T_5a52f_row1_col6\" class=\"data row1 col6\" >0:00:02</td>\n", + " <td id=\"T_5a52f_row1_col7\" class=\"data row1 col7\" >ok</td>\n", + " </tr>\n", + " </tbody></table>" + ], + "text/plain": [ + "<pandas.io.formats.style.Styler at 0x7fa37bdc43d0>" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/markdown": [ + "<br>HTML report saved as : [./logs/ci_report.html](./logs/ci_report.html)" + ], + "text/plain": [ + "<IPython.core.display.Markdown object>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import cookci\n", + "cookci.build_ci_report(report_name='./logs/ci_report_smart.json')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "<img width=\"80px\" src=\"../fidle/img/00-Fidle-logo-01.svg\"></img>" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.9" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/fidle/ci/smart.yml b/fidle/ci/basic_example.yml similarity index 56% rename from fidle/ci/smart.yml rename to fidle/ci/basic_example.yml index c1b6c73845e3bb762d97c411e3d6b709c1929f8a..d4316959df663c52f80a8474f4fa3784c3c827df 100644 --- a/fidle/ci/smart.yml +++ b/fidle/ci/basic_example.yml @@ -1,7 +1,8 @@ -config: +_metadata_: version: '1.0' output_tag: ==done== - save_figs: true + save_figs: false + description: A basic profile example LINR1: notebook_dir: LinearReg notebook_src: 01-Linear-Regression.ipynb @@ -17,13 +18,13 @@ PER57: # overrides: # scale: 0.1 # output_dir: ./data -GTSRB2: - notebook_dir: GTSRB - notebook_src: 02-First-convolutions.ipynb - notebook_out: default - overrides: - run_dir: ./run/GTSRB2_ci - dataset_name: set-24x24-L - batch_size: 32 - epochs: 3 - scale: 0.1 \ No newline at end of file +# GTSRB2: +# notebook_dir: GTSRB +# notebook_src: 02-First-convolutions.ipynb +# notebook_out: default +# overrides: +# run_dir: ./run/GTSRB2_ci +# dataset_name: set-24x24-L +# batch_size: 32 +# epochs: 3 +# scale: 0.1 \ No newline at end of file diff --git a/fidle/ci/default.yml b/fidle/ci/default.yml index 9853a84605bec890afc3aad6509bb74fd358e8cb..cd49f66c87019ed1405c4f2c6bd934b26dcde84f 100644 --- a/fidle/ci/default.yml +++ b/fidle/ci/default.yml @@ -1,7 +1,8 @@ -config: +_metadata_: version: '1.0' output_tag: ==done== save_figs: true + description: Default generated profile LINR1: notebook_dir: LinearReg notebook_src: 01-Linear-Regression.ipynb diff --git a/fidle/ci/smart_cpu.yml b/fidle/ci/smart_cpu.yml new file mode 100644 index 0000000000000000000000000000000000000000..c60263f54fd01f2677fe627785d4361101f52ca2 --- /dev/null +++ b/fidle/ci/smart_cpu.yml @@ -0,0 +1,98 @@ +_metadata_: + version: '1.0' + output_tag: ==done== + save_figs: true + description: Smart profile, for cpu +LINR1: + notebook_dir: LinearReg + notebook_src: 01-Linear-Regression.ipynb + notebook_out: default +GRAD1: + notebook_dir: LinearReg + notebook_src: 02-Gradient-descent.ipynb + notebook_out: default +POLR1: + notebook_dir: LinearReg + notebook_src: 03-Polynomial-Regression.ipynb + notebook_out: default +LOGR1: + notebook_dir: LinearReg + notebook_src: 04-Logistic-Regression.ipynb + notebook_out: default +PER57: + notebook_dir: IRIS + notebook_src: 01-Simple-Perceptron.ipynb + notebook_out: default +BHPD1: + notebook_dir: BHPD + notebook_src: 01-DNN-Regression.ipynb + notebook_out: default +BHPD2: + notebook_dir: BHPD + notebook_src: 02-DNN-Regression-Premium.ipynb + notebook_out: default +MNIST1: + notebook_dir: MNIST + notebook_src: 01-DNN-MNIST.ipynb + notebook_out: default +GTSRB1: + notebook_dir: GTSRB + notebook_src: 01-Preparation-of-data.ipynb + notebook_out: default + overrides: + scale: 0.05 + output_dir: ./data +GTSRB2: + notebook_dir: GTSRB + notebook_src: 02-First-convolutions.ipynb + notebook_out: default + overrides: + run_dir: ./run/GTSRB2_ci + dataset_name: set-24x24-L + batch_size: 64 + epochs: 5 + scale: 1 +GTSRB3: + notebook_dir: GTSRB + notebook_src: 03-Tracking-and-visualizing.ipynb + notebook_out: default + overrides: + run_dir: ./run/GTSRB3_ci + dataset_name: set-24x24-L + batch_size: 64 + epochs: 5 + scale: 1 +GTSRB4: + notebook_dir: GTSRB + notebook_src: 04-Data-augmentation.ipynb + notebook_out: default + overrides: + run_dir: ./run/GTSRB4_ci + dataset_name: set-24x24-L + batch_size: 64 + epochs: 5 + scale: 0.5 +GTSRB5: + notebook_dir: GTSRB + notebook_src: 05-Full-convolutions.ipynb + notebook_out: default + overrides: + run_dir: ./run/GTSRB5_ci + datasets: "['set-24x24-L', 'set-24x24-RGB', 'set-48x48-RGB']" + models: "{'v1':'get_model_v1', 'v2':'get_model_v2', 'v3':'get_model_v3'}" + batch_size: 64 + epochs: 5 + scale: 0.1 + with_datagen: True + verbose: 0 +GTSRB6: + notebook_dir: GTSRB + notebook_src: 06-Notebook-as-a-batch.ipynb + notebook_out: default +GTSRB7: + notebook_dir: GTSRB + notebook_src: 07-Show-report.ipynb + notebook_out: default + overrides: + run_dir: ./run/GTSRB7_ci + report_dir: ./run/GTSRB5 \ No newline at end of file diff --git a/fidle/cookci.py b/fidle/cookci.py index 6d7485e332fd5294c0d6dabe1594ba16191642c6..bb02183ff8d8aa25818d336491ad674048907c1c 100644 --- a/fidle/cookci.py +++ b/fidle/cookci.py @@ -20,15 +20,17 @@ from nbconvert.preprocessors import ExecutePreprocessor import re import yaml from collections import OrderedDict -from IPython.display import display +from IPython.display import display,Image,Markdown,HTML import pandas as pd sys.path.append('..') import fidle.config as config import fidle.cookindex as cookindex -start_time = None -end_time = None +VERSION = '1.0' + +start_time = {} +end_time = {} def get_ci_profile(catalog=None, output_tag='==done==', save_figs=True): ''' @@ -47,8 +49,11 @@ def get_ci_profile(catalog=None, output_tag='==done==', save_figs=True): if catalog is None: catalog = cookindex.read_catalog() - config = {'version':'1.0', 'output_tag':output_tag, 'save_figs':save_figs} - profile = { 'config':config } + metadata = { 'version' : '1.0', + 'output_tag' : output_tag, + 'save_figs' : save_figs, + 'description' : 'Default generated profile'} + profile = { '_metadata_':metadata } for id, about in catalog.items(): id = about['id'] @@ -73,7 +78,7 @@ def save_profile(profile, filename): '''Save profile in yaml format''' with open(filename,'wt') as fp: yaml.dump(profile, fp, sort_keys=False) - print(f'Catalog saved as {filename}') + print(f'Profile saved as {filename}') print('Entries : ',len(profile)-1) @@ -81,12 +86,12 @@ def load_profile(filename): '''Load yaml profile''' with open(filename,'r') as fp: profile=yaml.load(fp, Loader=yaml.FullLoader) - print(f'{filename} loaded.') - print('Entries : ',len(profile)-1) + print(f'\nLoad profile :{filename}') + print(' Entries : ',len(profile)-1) return profile -def run_profile(profile, top_dir='..'): +def run_profile(profile_name, report_name=None, top_dir='..'): ''' Récupère la liste des notebooks et des paramètres associés, décrit dans le profile, et pour chaque notebook : @@ -95,20 +100,37 @@ def run_profile(profile, top_dir='..'): Exécuter celui-ci Sauvegarder le notebook résultat, avec son nom taggé. Params: - profile : dict, profile d'éxécution + profile_name : nom du profile d'éxécution + report_name : Nom du rapport json généré top_dir : chemin relatif vers la racine fidle (..) ''' - # ---- My place - # - home = os.getcwd() + print('\nRun profile session - FIDLE 2021') + print(f'Version : {VERSION}') + + chrono_start('main') - # ---- Read profile + # ---- Retrieve profile # - config = profile['config'] + profile = load_profile(profile_name) + config = profile['_metadata_'] notebooks = profile - del notebooks['config'] + del notebooks['_metadata_'] + # ---- Create new ci_report + # + metadata = config + metadata['host'] = os.uname()[1] + metadata['profile'] = profile_name + if report_name is None: + report_name = config.CI_REPORT_JSON + report_name = os.path.abspath(report_name) + create_ci_report(report_name, metadata) + + # ---- My place + # + home = os.getcwd() + # ---- Save figs or not ? # os.environ['FIDLE_SAVE_FIGS']=str(config['save_figs']) @@ -157,9 +179,9 @@ def run_profile(profile, top_dir='..'): # ---- Top chrono # - chrono_start() - update_ci_report(id, notebook_dir, notebook_src, notebook_out, start=True) - + chrono_start('nb') + update_ci_report(report_name, id, notebook_dir, notebook_src, notebook_out, start=True) + # ---- Try to run... # print(' Run notebook...',end='') @@ -179,9 +201,9 @@ def run_profile(profile, top_dir='..'): # ---- Top chrono # - chrono_stop() - update_ci_report(id, notebook_dir, notebook_src, notebook_out, end=True, happy_end=happy_end) - print(' Duration : ',chrono_delay() ) + chrono_stop('nb') + update_ci_report(report_name, id, notebook_dir, notebook_src, notebook_out, end=True, happy_end=happy_end) + print(' Duration : ',chrono_get_delay('nb') ) # ---- Save notebook # @@ -193,43 +215,66 @@ def run_profile(profile, top_dir='..'): # os.chdir(home) for env_name in to_unset: - del os.environ[env_name] + del os.environ[env_name] + + # ---- End of running + chrono_stop('main') + print('\nEnd of running process') + print(' Duration :', chrono_get_delay('main')) + complete_ci_report(report_name) -def chrono_start(): +def chrono_start(id='default'): global start_time - start_time = datetime.datetime.now() - -def chrono_stop(): + start_time[id] = datetime.datetime.now() + +def chrono_stop(id='default'): global end_time - end_time = datetime.datetime.now() + end_time[id] = datetime.datetime.now() -def chrono_delay(in_seconds=False): +def chrono_get_delay(id='default', in_seconds=False): global start_time, end_time - delta = end_time - start_time + delta = end_time[id] - start_time[id] if in_seconds: return round(delta.total_seconds(),2) else: delta = delta - datetime.timedelta(microseconds=delta.microseconds) return str(delta) +def chrono_get_start(id='default'): + global start_time + return start_time[id].strftime("%d/%m/%y %H:%M:%S") + +def chrono_get_end(id='default'): + global end_time + return end_time[id].strftime("%d/%m/%y %H:%M:%S") +def reset_chrono(): + global start_time, end_time + start_time, end_time = {},{} + -def reset_ci_report(verbose=True): - data={} - with open(config.CI_REPORT_JSON,'wt') as fp: +def create_ci_report(filename, metadata, verbose=True): + metadata['start']=chrono_get_start('main') + data={ '_metadata_':metadata } + with open(filename,'wt') as fp: json.dump(data,fp,indent=4) - if verbose : print(f'Finished file has been reset.\n') - + if verbose : print(f'\nCreate new ci report : {filename}') -def update_ci_report(notebook_id, notebook_dir, notebook_src, notebook_out, start=False, end=False, happy_end=True): +def complete_ci_report(filename, verbose=True): + with open(filename) as fp: + report = json.load(fp) + report['_metadata_']['end'] = chrono_get_end('main') + report['_metadata_']['duration'] = chrono_get_delay('main') + with open(filename,'wt') as fp: + json.dump(report,fp,indent=4) + if verbose : print(f'\nComplete ci report : {filename}') + +def update_ci_report(filename, notebook_id, notebook_dir, notebook_src, notebook_out, start=False, end=False, happy_end=True): global start_time, end_time - - if not os.access(config.CI_REPORT_JSON, os.W_OK): - reset_ci_report(verbose=True) # ---- Load it - with open(config.CI_REPORT_JSON) as fp: + with open(filename) as fp: report = json.load(fp) # ---- Update as a start @@ -238,44 +283,51 @@ def update_ci_report(notebook_id, notebook_dir, notebook_src, notebook_out, star report[notebook_id]['dir'] = notebook_dir report[notebook_id]['src'] = notebook_src report[notebook_id]['out'] = notebook_out - report[notebook_id]['start'] = start_time.strftime("%d/%m/%y %H:%M:%S") + report[notebook_id]['start'] = chrono_get_start('nb') report[notebook_id]['end'] = '' report[notebook_id]['duration'] = 'Unfinished...' report[notebook_id]['state'] = 'Unfinished...' # ---- Update as an end if end is True: - report[notebook_id]['end'] = end_time.strftime("%d/%m/%y %H:%M:%S") - report[notebook_id]['duration'] = chrono_delay() + report[notebook_id]['end'] = chrono_get_end('nb') + report[notebook_id]['duration'] = chrono_get_delay('nb') report[notebook_id]['state'] = 'ok' if happy_end else 'ERROR' report[notebook_id]['out'] = notebook_out # changeg in case of error # ---- Save it - with open(config.CI_REPORT_JSON,'wt') as fp: + with open(filename,'wt') as fp: json.dump(report,fp,indent=4) -def build_ci_report(display_output=True, save_html=True): +def build_ci_report(report_name=None, display_output=True, save_html=True): # ---- Load ci report # - with open(config.CI_REPORT_JSON) as infile: + if report_name is None: + report_name = config.CI_REPORT_JSON + with open(report_name) as infile: ci_report = json.load( infile ) + # ---- metadata + # + metadata=ci_report['_metadata_'] + del ci_report['_metadata_'] + + metadata_md='' + metadata_html='' + for name,value in metadata.items(): + metadata_md += f'**{name.title()}** : {value} \n' + metadata_html += f'<b>{name.title()}</b> : {value} <br>\n' + # ---- Create a nice DataFrame # df=pd.DataFrame(ci_report) df=df.transpose() df = df.rename_axis('id').reset_index() - - # ---- Change text columns, for nice html links - # - df['id'] = df.apply(lambda r: f"<a href='../{r['dir']}/{r['src']}'>{r['id']}</a>", axis=1) - df['src'] = df.apply(lambda r: f"<a href='../{r['dir']}/{r['src']}'>{r['src']}</a>", axis=1) - df['out'] = df.apply(lambda r: f"<a href='../{r['dir']}/{r['out']}'>{r['out']}</a>", axis=1) - - # ---- Add styles to be nice + + # ---- Few styles to be nice # styles = [ dict(selector="td", props=[("font-size", "110%"), ("text-align", "left")]), @@ -284,42 +336,60 @@ def build_ci_report(display_output=True, save_html=True): def still_pending(v): return 'background-color: OrangeRed; color:white' if v == 'ERROR' else '' - output = df[df.columns.values].style.set_table_styles(styles).hide_index().applymap(still_pending) + # ---- Links version : display + # + if display_output: + + ddf=df.copy() + ddf['id'] = ddf.apply(lambda r: f"<a href='../{r['dir']}/{r['src']}'>{r['id']}</a>", axis=1) + ddf['src'] = ddf.apply(lambda r: f"<a href='../{r['dir']}/{r['src']}'>{r['src']}</a>", axis=1) + ddf['out'] = ddf.apply(lambda r: f"<a href='../{r['dir']}/{r['out']}'>{r['out']}</a>", axis=1) + ddf.columns = [x.title() for x in ddf.columns] + + output = ddf[ddf.columns.values].style.set_table_styles(styles).hide_index().applymap(still_pending) + display(Markdown('### About :')) + display(Markdown(metadata_md)) + display(Markdown('### Details :')) + display(output) - # ---- html report + # ---- Basic version : html report # if save_html: - html = _get_html_report(output) + + df.columns = [x.title() for x in df.columns] + output = df[df.columns.values].style.set_table_styles(styles).hide_index().applymap(still_pending) + + html = _get_html_report(metadata_html, output) with open(config.CI_REPORT_HTML, "wt") as fp: fp.write(html) - - # ---- display output - # - if display_output: - display(output) - + display(Markdown('<br>HTML report saved as : [./logs/ci_report.html](./logs/ci_report.html)')) + -def _get_html_report(output): +def _get_html_report(metadata_html, output): with open('./img/00-Fidle-logo-01-80px.svg','r') as fp: logo = fp.read() - now = datetime.datetime.now().strftime("%A %-d %B %Y, %H:%M:%S") html = f"""\ <html> <head><title>FIDLE - CI Report</title></head> + <body> <style> body{{ font-family: sans-serif; }} + div.title{{ + font-size: 1.2em; + font-weight: bold; + padding: 15px 0px 10px 0px; }} a{{ color: SteelBlue; text-decoration:none; }} table{{ border-collapse : collapse; - font-size : 80% + font-size : 0.9em; }} td{{ border-style: solid; @@ -327,16 +397,15 @@ def _get_html_report(output): border-color: lightgrey; padding: 5px; }} - .header{{ padding:20px 0px 0px 30px; }} - .result{{ padding:10px 0px 20px 30px; }} + .metadata{{ padding: 10px 0px 10px 30px; font-size: 0.9em; }} + .result{{ padding: 10px 0px 10px 30px; }} </style> - <body> <br>Hi, <p>Below is the result of the continuous integration tests of the Fidle project:</p> - <div class="header"><b>Report date :</b> {now}</div> - <div class="result"> - {output.render()} - </div> + <div class='title'>About :</div> + <div class="metadata">{metadata_html}</div> + <div class='title'>Details :</div> + <div class="result">{output.render()}</div> {logo} diff --git a/fidle/logs/ci_report.html b/fidle/logs/ci_report.html index 323f0ce1b5318f139c5035cf0a76b25d954f0505..08321259eddc21b04b94db33dbf028cc82fe511d 100644 --- a/fidle/logs/ci_report.html +++ b/fidle/logs/ci_report.html @@ -1,16 +1,21 @@ <html> <head><title>FIDLE - CI Report</title></head> + <body> <style> body{ font-family: sans-serif; } + div.title{ + font-size: 1.2em; + font-weight: bold; + padding: 15px 0px 10px 0px; } a{ color: SteelBlue; text-decoration:none; } table{ border-collapse : collapse; - font-size : 80% + font-size : 0.9em; } td{ border-style: solid; @@ -18,57 +23,130 @@ border-color: lightgrey; padding: 5px; } - .header{ padding:20px 0px 0px 30px; } - .result{ padding:10px 0px 20px 30px; } + .metadata{ padding: 10px 0px 10px 30px; font-size: 0.9em; } + .result{ padding: 10px 0px 10px 30px; } </style> - <body> <br>Hi, <p>Below is the result of the continuous integration tests of the Fidle project:</p> - <div class="header"><b>Report date :</b> Monday 11 January 2021, 01:07:55</div> - <div class="result"> - <style type="text/css" > - #T_58cf4_ td { + <div class='title'>About :</div> + <div class="metadata"><b>Version</b> : 1.0 <br> +<b>Output_Tag</b> : ==done== <br> +<b>Save_Figs</b> : True <br> +<b>Description</b> : Smart profile, for cpu <br> +<b>Host</b> : Oban <br> +<b>Profile</b> : ./ci/smart_cpu.yml <br> +<b>Start</b> : 11/01/21 16:38:18 <br> +</div> + <div class='title'>Details :</div> + <div class="result"><style type="text/css" > + #T_28761_ td { font-size: 110%; text-align: left; - } #T_58cf4_ th { + } #T_28761_ th { font-size: 110%; text-align: left; - }#T_58cf4_row2_col7{ - background-color: OrangeRed; - color: white; - }</style><table id="T_58cf4_" ><thead> <tr> <th class="col_heading level0 col0" >id</th> <th class="col_heading level0 col1" >dir</th> <th class="col_heading level0 col2" >src</th> <th class="col_heading level0 col3" >out</th> <th class="col_heading level0 col4" >start</th> <th class="col_heading level0 col5" >end</th> <th class="col_heading level0 col6" >duration</th> <th class="col_heading level0 col7" >state</th> </tr></thead><tbody> + }</style><table id="T_28761_" ><thead> <tr> <th class="col_heading level0 col0" >Id</th> <th class="col_heading level0 col1" >Dir</th> <th class="col_heading level0 col2" >Src</th> <th class="col_heading level0 col3" >Out</th> <th class="col_heading level0 col4" >Start</th> <th class="col_heading level0 col5" >End</th> <th class="col_heading level0 col6" >Duration</th> <th class="col_heading level0 col7" >State</th> </tr></thead><tbody> <tr> - <td id="T_58cf4_row0_col0" class="data row0 col0" ><a href='../LinearReg/01-Linear-Regression.ipynb'>LINR1</a></td> - <td id="T_58cf4_row0_col1" class="data row0 col1" >LinearReg</td> - <td id="T_58cf4_row0_col2" class="data row0 col2" ><a href='../LinearReg/01-Linear-Regression.ipynb'>01-Linear-Regression.ipynb</a></td> - <td id="T_58cf4_row0_col3" class="data row0 col3" ><a href='../LinearReg/01-Linear-Regression==done==.ipynb'>01-Linear-Regression==done==.ipynb</a></td> - <td id="T_58cf4_row0_col4" class="data row0 col4" >11/01/21 01:07:35</td> - <td id="T_58cf4_row0_col5" class="data row0 col5" >11/01/21 01:07:39</td> - <td id="T_58cf4_row0_col6" class="data row0 col6" >0:00:04</td> - <td id="T_58cf4_row0_col7" class="data row0 col7" >ok</td> + <td id="T_28761_row0_col0" class="data row0 col0" >LINR1</td> + <td id="T_28761_row0_col1" class="data row0 col1" >LinearReg</td> + <td id="T_28761_row0_col2" class="data row0 col2" >01-Linear-Regression.ipynb</td> + <td id="T_28761_row0_col3" class="data row0 col3" >01-Linear-Regression==done==.ipynb</td> + <td id="T_28761_row0_col4" class="data row0 col4" >11/01/21 16:38:18</td> + <td id="T_28761_row0_col5" class="data row0 col5" >11/01/21 16:38:21</td> + <td id="T_28761_row0_col6" class="data row0 col6" >0:00:03</td> + <td id="T_28761_row0_col7" class="data row0 col7" >ok</td> + </tr> + <tr> + <td id="T_28761_row1_col0" class="data row1 col0" >GRAD1</td> + <td id="T_28761_row1_col1" class="data row1 col1" >LinearReg</td> + <td id="T_28761_row1_col2" class="data row1 col2" >02-Gradient-descent.ipynb</td> + <td id="T_28761_row1_col3" class="data row1 col3" >02-Gradient-descent==done==.ipynb</td> + <td id="T_28761_row1_col4" class="data row1 col4" >11/01/21 16:38:21</td> + <td id="T_28761_row1_col5" class="data row1 col5" >11/01/21 16:38:28</td> + <td id="T_28761_row1_col6" class="data row1 col6" >0:00:06</td> + <td id="T_28761_row1_col7" class="data row1 col7" >ok</td> + </tr> + <tr> + <td id="T_28761_row2_col0" class="data row2 col0" >POLR1</td> + <td id="T_28761_row2_col1" class="data row2 col1" >LinearReg</td> + <td id="T_28761_row2_col2" class="data row2 col2" >03-Polynomial-Regression.ipynb</td> + <td id="T_28761_row2_col3" class="data row2 col3" >03-Polynomial-Regression==done==.ipynb</td> + <td id="T_28761_row2_col4" class="data row2 col4" >11/01/21 16:38:28</td> + <td id="T_28761_row2_col5" class="data row2 col5" >11/01/21 16:38:31</td> + <td id="T_28761_row2_col6" class="data row2 col6" >0:00:03</td> + <td id="T_28761_row2_col7" class="data row2 col7" >ok</td> + </tr> + <tr> + <td id="T_28761_row3_col0" class="data row3 col0" >LOGR1</td> + <td id="T_28761_row3_col1" class="data row3 col1" >LinearReg</td> + <td id="T_28761_row3_col2" class="data row3 col2" >04-Logistic-Regression.ipynb</td> + <td id="T_28761_row3_col3" class="data row3 col3" >04-Logistic-Regression==done==.ipynb</td> + <td id="T_28761_row3_col4" class="data row3 col4" >11/01/21 16:38:31</td> + <td id="T_28761_row3_col5" class="data row3 col5" >11/01/21 16:38:35</td> + <td id="T_28761_row3_col6" class="data row3 col6" >0:00:03</td> + <td id="T_28761_row3_col7" class="data row3 col7" >ok</td> + </tr> + <tr> + <td id="T_28761_row4_col0" class="data row4 col0" >PER57</td> + <td id="T_28761_row4_col1" class="data row4 col1" >IRIS</td> + <td id="T_28761_row4_col2" class="data row4 col2" >01-Simple-Perceptron.ipynb</td> + <td id="T_28761_row4_col3" class="data row4 col3" >01-Simple-Perceptron==done==.ipynb</td> + <td id="T_28761_row4_col4" class="data row4 col4" >11/01/21 16:38:35</td> + <td id="T_28761_row4_col5" class="data row4 col5" >11/01/21 16:38:38</td> + <td id="T_28761_row4_col6" class="data row4 col6" >0:00:02</td> + <td id="T_28761_row4_col7" class="data row4 col7" >ok</td> + </tr> + <tr> + <td id="T_28761_row5_col0" class="data row5 col0" >BHPD1</td> + <td id="T_28761_row5_col1" class="data row5 col1" >BHPD</td> + <td id="T_28761_row5_col2" class="data row5 col2" >01-DNN-Regression.ipynb</td> + <td id="T_28761_row5_col3" class="data row5 col3" >01-DNN-Regression==done==.ipynb</td> + <td id="T_28761_row5_col4" class="data row5 col4" >11/01/21 16:38:38</td> + <td id="T_28761_row5_col5" class="data row5 col5" >11/01/21 16:38:51</td> + <td id="T_28761_row5_col6" class="data row5 col6" >0:00:13</td> + <td id="T_28761_row5_col7" class="data row5 col7" >ok</td> + </tr> + <tr> + <td id="T_28761_row6_col0" class="data row6 col0" >BHPD2</td> + <td id="T_28761_row6_col1" class="data row6 col1" >BHPD</td> + <td id="T_28761_row6_col2" class="data row6 col2" >02-DNN-Regression-Premium.ipynb</td> + <td id="T_28761_row6_col3" class="data row6 col3" >02-DNN-Regression-Premium==done==.ipynb</td> + <td id="T_28761_row6_col4" class="data row6 col4" >11/01/21 16:38:51</td> + <td id="T_28761_row6_col5" class="data row6 col5" >11/01/21 16:39:04</td> + <td id="T_28761_row6_col6" class="data row6 col6" >0:00:13</td> + <td id="T_28761_row6_col7" class="data row6 col7" >ok</td> + </tr> + <tr> + <td id="T_28761_row7_col0" class="data row7 col0" >MNIST1</td> + <td id="T_28761_row7_col1" class="data row7 col1" >MNIST</td> + <td id="T_28761_row7_col2" class="data row7 col2" >01-DNN-MNIST.ipynb</td> + <td id="T_28761_row7_col3" class="data row7 col3" >01-DNN-MNIST==done==.ipynb</td> + <td id="T_28761_row7_col4" class="data row7 col4" >11/01/21 16:39:04</td> + <td id="T_28761_row7_col5" class="data row7 col5" >11/01/21 16:39:42</td> + <td id="T_28761_row7_col6" class="data row7 col6" >0:00:37</td> + <td id="T_28761_row7_col7" class="data row7 col7" >ok</td> </tr> <tr> - <td id="T_58cf4_row1_col0" class="data row1 col0" ><a href='../IRIS/01-Simple-Perceptron.ipynb'>PER57</a></td> - <td id="T_58cf4_row1_col1" class="data row1 col1" >IRIS</td> - <td id="T_58cf4_row1_col2" class="data row1 col2" ><a href='../IRIS/01-Simple-Perceptron.ipynb'>01-Simple-Perceptron.ipynb</a></td> - <td id="T_58cf4_row1_col3" class="data row1 col3" ><a href='../IRIS/01-Simple-Perceptron==done==.ipynb'>01-Simple-Perceptron==done==.ipynb</a></td> - <td id="T_58cf4_row1_col4" class="data row1 col4" >11/01/21 01:07:39</td> - <td id="T_58cf4_row1_col5" class="data row1 col5" >11/01/21 01:07:42</td> - <td id="T_58cf4_row1_col6" class="data row1 col6" >0:00:02</td> - <td id="T_58cf4_row1_col7" class="data row1 col7" >ok</td> + <td id="T_28761_row8_col0" class="data row8 col0" >GTSRB1</td> + <td id="T_28761_row8_col1" class="data row8 col1" >GTSRB</td> + <td id="T_28761_row8_col2" class="data row8 col2" >01-Preparation-of-data.ipynb</td> + <td id="T_28761_row8_col3" class="data row8 col3" >01-Preparation-of-data==done==.ipynb</td> + <td id="T_28761_row8_col4" class="data row8 col4" >11/01/21 16:39:42</td> + <td id="T_28761_row8_col5" class="data row8 col5" >11/01/21 16:41:06</td> + <td id="T_28761_row8_col6" class="data row8 col6" >0:01:24</td> + <td id="T_28761_row8_col7" class="data row8 col7" >ok</td> </tr> <tr> - <td id="T_58cf4_row2_col0" class="data row2 col0" ><a href='../GTSRB/02-First-convolutions.ipynb'>GTSRB2</a></td> - <td id="T_58cf4_row2_col1" class="data row2 col1" >GTSRB</td> - <td id="T_58cf4_row2_col2" class="data row2 col2" ><a href='../GTSRB/02-First-convolutions.ipynb'>02-First-convolutions.ipynb</a></td> - <td id="T_58cf4_row2_col3" class="data row2 col3" ><a href='../GTSRB/02-First-convolutions==ERROR==.ipynb'>02-First-convolutions==ERROR==.ipynb</a></td> - <td id="T_58cf4_row2_col4" class="data row2 col4" >11/01/21 01:07:42</td> - <td id="T_58cf4_row2_col5" class="data row2 col5" >11/01/21 01:07:44</td> - <td id="T_58cf4_row2_col6" class="data row2 col6" >0:00:02</td> - <td id="T_58cf4_row2_col7" class="data row2 col7" >ERROR</td> + <td id="T_28761_row9_col0" class="data row9 col0" >GTSRB2</td> + <td id="T_28761_row9_col1" class="data row9 col1" >GTSRB</td> + <td id="T_28761_row9_col2" class="data row9 col2" >02-First-convolutions.ipynb</td> + <td id="T_28761_row9_col3" class="data row9 col3" >02-First-convolutions==done==.ipynb</td> + <td id="T_28761_row9_col4" class="data row9 col4" >11/01/21 16:41:06</td> + <td id="T_28761_row9_col5" class="data row9 col5" ></td> + <td id="T_28761_row9_col6" class="data row9 col6" >Unfinished...</td> + <td id="T_28761_row9_col7" class="data row9 col7" >Unfinished...</td> </tr> - </tbody></table> - </div> + </tbody></table></div> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 140.2164 40.848" width="80px"><title>00-Fidle-logo-01</title><g id="Calque_2" data-name="Calque 2"><g id="Calque_4" data-name="Calque 4"><path d="M46.1913,31.74a12.9222,12.9222,0,0,0,5.2755-1.77,6.4763,6.4763,0,0,1,2.3629-.9239,14.6364,14.6364,0,0,0-5.7616-16.4446,17.0565,17.0565,0,0,0-11.8732-2.0051c-4.1719.69-8.4957,3.8461-10.189,5.2622-1.0189.8536-13.1385,12.3424-18.1936,10.0527-3.42-1.5492,2.6862-7.1873-.1144-12.3393a.2236.2236,0,0,0-.373-.0248c-1.4257,1.9233-2.8193,4.2317-4.7179,3.1953-.8482-.4632-1.6116-1.9422-2.2-2.8775A.2216.2216,0,0,0,0,13.9917,23.35,23.35,0,0,0,5.87,28.2417a35.3776,35.3776,0,0,0,24.34,12.518c5.3439.5321,18.0193-1.1527,23.0835-10.2646a12.7681,12.7681,0,0,0-1.2217.6066,14.2177,14.2177,0,0,1-5.7629,1.9167c-.1761.0163-.3511.0236-.5261.0236a10.1733,10.1733,0,0,1-5.7446-2.303,1.0764,1.0764,0,1,1,.8227-1.0443c0,.0176-.0042.0339-.0054.0515C41.8966,30.5423,44.0669,31.9474,46.1913,31.74ZM30.0385,36.5091a19.6093,19.6093,0,0,1-4.6162.8385c-1.0425.0006-1.476-.2954-1.6824-.7392-.5431-1.1678,1.4136-2.8563,3.1493-4.0677a.6418.6418,0,1,1,.7343,1.0528,10.5781,10.5781,0,0,0-2.651,2.4368c.339.0732,1.44.12,4.733-.7616a.6422.6422,0,0,1,.333,1.24Zm14.87-15.6442a2.4512,2.4512,0,0,1,2.38,2.3617,1.6015,1.6015,0,1,0-1.4179,2.34,1.6573,1.6573,0,0,0,.2973-.03,2.28,2.28,0,0,1-1.2593.3875,2.5337,2.5337,0,0,1,0-5.06ZM36.6423,4.436A1.2835,1.2835,0,0,0,37.1466,6.18c.6211.342,1.9294-.402,2.7231.7071.4122.5763-.8627-2.6129-1.4839-2.9556A1.2827,1.2827,0,0,0,36.6423,4.436Zm6.5389.1374c-1.5995.9378-1.8961,4.8154-1.4838,4.2391a7.2989,7.2989,0,0,1,2.7231-1.9906,1.2837,1.2837,0,0,0-1.2393-2.2485ZM41.5587.2981c-.8179.9462-.2579,3.4-.1114,2.95a5.2169,5.2169,0,0,1,1.3174-1.8537A.8415.8415,0,0,0,42.7441.2054.8332.8332,0,0,0,41.5587.2981Z" style="fill:#e12229"/><path d="M65.6671,13.7493H77.3946V15.158H67.3223v9.4379h9.2271v1.4087H67.3223v11.481H65.6671Z" style="fill:#808285"/><path d="M83.5909,13.7493V37.4856H81.9356V13.7493Z" style="fill:#808285"/><path d="M89.3658,14.0662a39.0353,39.0353,0,0,1,6.0576-.4932c4.3316,0,7.607,1.1621,9.5791,3.24a11.2256,11.2256,0,0,1,2.958,8.2056,13.0738,13.0738,0,0,1-3.0991,9.0156c-2.1128,2.2891-5.67,3.6275-10.248,3.6275a50.7148,50.7148,0,0,1-5.2476-.2115Zm1.6553,22.0107a29.8576,29.8576,0,0,0,3.8388.1763c7.607,0,11.375-4.2617,11.375-11.1289.0352-6.022-3.31-10.1426-10.9174-10.1426a25.2377,25.2377,0,0,0-4.2964.352Z" style="fill:#808285"/><path d="M112.15,13.7493h1.6553V36.0769h10.6006v1.4087H112.15Z" style="fill:#808285"/><path d="M139.0894,25.6877h-9.5088V36.0769h10.6358v1.4087h-12.291V13.7493h11.7275V15.158H129.5806v9.1211h9.5088Z" style="fill:#808285"/></g></g></svg> diff --git a/fidle/logs/ci_report.json b/fidle/logs/ci_report.json deleted file mode 100644 index 2e8a6868272449308b7466607c0a26f238d1aae7..0000000000000000000000000000000000000000 --- a/fidle/logs/ci_report.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "LINR1": { - "dir": "LinearReg", - "src": "01-Linear-Regression.ipynb", - "out": "01-Linear-Regression==done==.ipynb", - "start": "11/01/21 01:07:35", - "end": "11/01/21 01:07:39", - "duration": "0:00:04", - "state": "ok" - }, - "PER57": { - "dir": "IRIS", - "src": "01-Simple-Perceptron.ipynb", - "out": "01-Simple-Perceptron==done==.ipynb", - "start": "11/01/21 01:07:39", - "end": "11/01/21 01:07:42", - "duration": "0:00:02", - "state": "ok" - }, - "GTSRB2": { - "dir": "GTSRB", - "src": "02-First-convolutions.ipynb", - "out": "02-First-convolutions==ERROR==.ipynb", - "start": "11/01/21 01:07:42", - "end": "11/01/21 01:07:44", - "duration": "0:00:02", - "state": "ERROR" - } -} \ No newline at end of file diff --git a/fidle/logs/ci_report_smart.json b/fidle/logs/ci_report_smart.json new file mode 100644 index 0000000000000000000000000000000000000000..100f922c95cf1bfd2fff469bd726f97ccb3865fc --- /dev/null +++ b/fidle/logs/ci_report_smart.json @@ -0,0 +1,148 @@ +{ + "_metadata_": { + "version": "1.0", + "output_tag": "==done==", + "save_figs": true, + "description": "Smart profile, for cpu", + "host": "Oban", + "profile": "./ci/smart_cpu.yml", + "start": "11/01/21 16:38:18", + "end": "11/01/21 16:51:47", + "duration": "0:13:29" + }, + "LINR1": { + "dir": "LinearReg", + "src": "01-Linear-Regression.ipynb", + "out": "01-Linear-Regression==done==.ipynb", + "start": "11/01/21 16:38:18", + "end": "11/01/21 16:38:21", + "duration": "0:00:03", + "state": "ok" + }, + "GRAD1": { + "dir": "LinearReg", + "src": "02-Gradient-descent.ipynb", + "out": "02-Gradient-descent==done==.ipynb", + "start": "11/01/21 16:38:21", + "end": "11/01/21 16:38:28", + "duration": "0:00:06", + "state": "ok" + }, + "POLR1": { + "dir": "LinearReg", + "src": "03-Polynomial-Regression.ipynb", + "out": "03-Polynomial-Regression==done==.ipynb", + "start": "11/01/21 16:38:28", + "end": "11/01/21 16:38:31", + "duration": "0:00:03", + "state": "ok" + }, + "LOGR1": { + "dir": "LinearReg", + "src": "04-Logistic-Regression.ipynb", + "out": "04-Logistic-Regression==done==.ipynb", + "start": "11/01/21 16:38:31", + "end": "11/01/21 16:38:35", + "duration": "0:00:03", + "state": "ok" + }, + "PER57": { + "dir": "IRIS", + "src": "01-Simple-Perceptron.ipynb", + "out": "01-Simple-Perceptron==done==.ipynb", + "start": "11/01/21 16:38:35", + "end": "11/01/21 16:38:38", + "duration": "0:00:02", + "state": "ok" + }, + "BHPD1": { + "dir": "BHPD", + "src": "01-DNN-Regression.ipynb", + "out": "01-DNN-Regression==done==.ipynb", + "start": "11/01/21 16:38:38", + "end": "11/01/21 16:38:51", + "duration": "0:00:13", + "state": "ok" + }, + "BHPD2": { + "dir": "BHPD", + "src": "02-DNN-Regression-Premium.ipynb", + "out": "02-DNN-Regression-Premium==done==.ipynb", + "start": "11/01/21 16:38:51", + "end": "11/01/21 16:39:04", + "duration": "0:00:13", + "state": "ok" + }, + "MNIST1": { + "dir": "MNIST", + "src": "01-DNN-MNIST.ipynb", + "out": "01-DNN-MNIST==done==.ipynb", + "start": "11/01/21 16:39:04", + "end": "11/01/21 16:39:42", + "duration": "0:00:37", + "state": "ok" + }, + "GTSRB1": { + "dir": "GTSRB", + "src": "01-Preparation-of-data.ipynb", + "out": "01-Preparation-of-data==done==.ipynb", + "start": "11/01/21 16:39:42", + "end": "11/01/21 16:41:06", + "duration": "0:01:24", + "state": "ok" + }, + "GTSRB2": { + "dir": "GTSRB", + "src": "02-First-convolutions.ipynb", + "out": "02-First-convolutions==done==.ipynb", + "start": "11/01/21 16:41:06", + "end": "11/01/21 16:45:02", + "duration": "0:03:55", + "state": "ok" + }, + "GTSRB3": { + "dir": "GTSRB", + "src": "03-Tracking-and-visualizing.ipynb", + "out": "03-Tracking-and-visualizing==done==.ipynb", + "start": "11/01/21 16:45:02", + "end": "11/01/21 16:49:25", + "duration": "0:04:22", + "state": "ok" + }, + "GTSRB4": { + "dir": "GTSRB", + "src": "04-Data-augmentation.ipynb", + "out": "04-Data-augmentation==ERROR==.ipynb", + "start": "11/01/21 16:49:26", + "end": "11/01/21 16:51:38", + "duration": "0:02:12", + "state": "ERROR" + }, + "GTSRB5": { + "dir": "GTSRB", + "src": "05-Full-convolutions.ipynb", + "out": "05-Full-convolutions==ERROR==.ipynb", + "start": "11/01/21 16:51:39", + "end": "11/01/21 16:51:42", + "duration": "0:00:02", + "state": "ERROR" + }, + "GTSRB6": { + "dir": "GTSRB", + "src": "06-Notebook-as-a-batch.ipynb", + "out": "06-Notebook-as-a-batch==done==.ipynb", + "start": "11/01/21 16:51:42", + "end": "11/01/21 16:51:45", + "duration": "0:00:03", + "state": "ok" + }, + "GTSRB7": { + "dir": "GTSRB", + "src": "07-Show-report.ipynb", + "out": "07-Show-report==done==.ipynb", + "start": "11/01/21 16:51:45", + "end": "11/01/21 16:51:47", + "duration": "0:00:02", + "state": "ok" + } +} \ No newline at end of file