Skip to content
Snippets Groups Projects

Add data augmentation

parent d5434aea
No related branches found
No related tags found
No related merge requests found
%% Cell type:markdown id: tags:
German Traffic Sign Recognition Benchmark (GTSRB)
=================================================
---
Introduction au Deep Learning (IDLE) - S. Aria, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020
## Episode 3 : Tracking, visualizing and save models
Our main steps:
- Monitoring and understanding our model training
- Add recovery points
- Analyze the results
- Restore and run recovery pont
## 1/ Import and init
%% Cell type:code id: tags:
``` python
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.callbacks import TensorBoard
import numpy as np
import h5py
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sn
import os, time, random
import idle.pwk as ooo
from importlib import reload
ooo.init()
```
%% Cell type:markdown id: tags:
## 2/ Load dataset
Dataset is one of the saved dataset: RGB25, RGB35, L25, L35, etc.
First of all, we're going to use a smart dataset : **set-24x24-L**
(with a GPU, it only takes 35'' compared to more than 5' with a CPU !)
%% Cell type:code id: tags:
``` python
%%time
def read_dataset(name):
'''Reads h5 dataset from ./data
Arguments: dataset name, without .h5
Returns: x_train,y_train,x_test,y_test data'''
# ---- Read dataset
filename='./data/'+name+'.h5'
with h5py.File(filename) as f:
x_train = f['x_train'][:]
y_train = f['y_train'][:]
x_test = f['x_test'][:]
y_test = f['y_test'][:]
# ---- done
print('Dataset "{}" is loaded. ({:.1f} Mo)\n'.format(name,os.path.getsize(filename)/(1024*1024)))
return x_train,y_train,x_test,y_test
x_train,y_train,x_test,y_test = read_dataset('set-24x24-L')
```
%% Cell type:markdown id: tags:
## 3/ Have a look to the dataset
Note: Data must be reshape for matplotlib
%% Cell type:code id: tags:
``` python
print("x_train : ", x_train.shape)
print("y_train : ", y_train.shape)
print("x_test : ", x_test.shape)
print("y_test : ", y_test.shape)
ooo.plot_images(x_train, y_train, range(12), columns=6, x_size=2, y_size=2)
ooo.plot_images(x_train, y_train, range(36), columns=12, x_size=1, y_size=1)
```
%% Cell type:markdown id: tags:
## 4/ Create model
We will now build a model and train it...
Some models...
%% Cell type:code id: tags:
``` python
# A basic model
#
def get_model_v1(lx,ly,lz):
model = keras.models.Sequential()
model.add( keras.layers.Conv2D(96, (3,3), activation='relu', input_shape=(lx,ly,lz)))
model.add( keras.layers.MaxPooling2D((2, 2)))
model.add( keras.layers.Dropout(0.2))
model.add( keras.layers.Conv2D(192, (3, 3), activation='relu'))
model.add( keras.layers.MaxPooling2D((2, 2)))
model.add( keras.layers.Dropout(0.2))
model.add( keras.layers.Flatten())
model.add( keras.layers.Dense(1500, activation='relu'))
model.add( keras.layers.Dropout(0.5))
model.add( keras.layers.Dense(43, activation='softmax'))
return model
```
%% Cell type:markdown id: tags:
## 5/ Prepare callbacks
We will add 2 callbacks :
- **TensorBoard**
Training logs, which can be visualised with Tensorboard.
`#tensorboard --logdir ./run/logs`
IMPORTANT : Relancer tensorboard à chaque run
- **Model backup**
It is possible to save the model each xx epoch or at each improvement.
The model can be saved completely or partially (weight).
For full format, we can use HDF5 format.
%% Cell type:code id: tags:
``` python
%%bash
# To clean old logs and saved model, run this cell
#
#/bin/rm -r ./run/logs 2>/dev/null
#/bin/rm -r ./run/models 2>/dev/null
/bin/ls -l ./run 2>/dev/null
```
%% Cell type:code id: tags:
``` python
ooo.mkdir('./run/models')
ooo.mkdir('./run/logs')
# ---- Callback tensorboard
log_dir = "./run/logs/tb_" + ooo.tag_now()
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
# ---- Callback ModelCheckpoint - Save best model
save_dir = "./run/models/best-model.h5"
bestmodel_callback = tf.keras.callbacks.ModelCheckpoint(filepath=save_dir, verbose=0, monitor='accuracy', save_best_only=True)
# ---- Callback ModelCheckpoint - Save model each epochs
save_dir = "./run/models/model-{epoch:04d}.h5"
savemodel_callback = tf.keras.callbacks.ModelCheckpoint(filepath=save_dir, verbose=0, save_freq=2000*5)
```
%% Cell type:markdown id: tags:
## 5/ Train the model
**Get the shape of my data :**
%% Cell type:code id: tags:
``` python
(n,lx,ly,lz) = x_train.shape
print("Images of the dataset have this folowing shape : ",(lx,ly,lz))
```
%% Cell type:markdown id: tags:
**Get and compile a model, with the data shape :**
%% Cell type:code id: tags:
``` python
model = get_model_v1(lx,ly,lz)
# model.summary()
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
```
%% Cell type:markdown id: tags:
**Train it :**
Note : La courbe d'apprentissage est visible en temps réel avec Tensorboard :
`#tensorboard --logdir ./run/logs`
%% Cell type:code id: tags:
``` python
%%time
batch_size = 64
epochs = 30
# ---- Shuffle train data
x_train,y_train=ooo.shuffle_np_dataset(x_train,y_train)
# ---- Train
# Note: To be faster in our example, we can take only 2000 values
#
history = model.fit( x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test[:200], y_test[:200]),
callbacks=[tensorboard_callback, bestmodel_callback, savemodel_callback] )
model.save('./run/models/last-model.h5')
```
%% Cell type:markdown id: tags:
**Evaluate it :**
%% Cell type:code id: tags:
``` python
max_val_accuracy = max(history.history["val_accuracy"])
print("Max validation accuracy is : {:.4f}".format(max_val_accuracy))
```
%% Cell type:code id: tags:
``` python
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss : {:5.4f}'.format(score[0]))
print('Test accuracy : {:5.4f}'.format(score[1]))
```
%% Cell type:markdown id: tags:
## 6/ History
The return of model.fit() returns us the learning history
%% Cell type:code id: tags:
``` python
ooo.plot_history(history)
```
%% Cell type:markdown id: tags:
## 7/ Evaluation and confusion
%% Cell type:code id: tags:
``` python
reload(ooo)
y_pred = model.predict_classes(x_test)
conf_mat = confusion_matrix(y_test,y_pred, normalize="true", labels=range(43))
ooo.plot_confusion_matrix(conf_mat)
```
%% Cell type:markdown id: tags:
## 8/ Restore and evaluate
### 8.1/ List saved models :
%% Cell type:code id: tags:
``` python
!find ./run/models/
```
%% Cell type:markdown id: tags:
### 8.2/ Restore a model :
%% Cell type:code id: tags:
``` python
loaded_model = tf.keras.models.load_model('./run/models/best-model.h5')
# best_model.summary()
print("Loaded.")
```
%% Cell type:markdown id: tags:
### 8.3/ Evaluate it :
%% Cell type:code id: tags:
``` python
score = loaded_model.evaluate(x_test, y_test, verbose=0)
print('Test loss : {:5.4f}'.format(score[0]))
print('Test accuracy : {:5.4f}'.format(score[1]))
```
%% Cell type:markdown id: tags:
### 8.4/ Make a prediction :
%% Cell type:code id: tags:
``` python
# ---- Get a random image
#
i = random.randint(1,len(x_test))
x,y = x_test[i], y_test[i]
# ---- Do prediction
#
predictions = loaded_model.predict( np.array([x]) )
# ---- A prediction is just the output layer
#
print("\nOutput layer from model is (x100) :\n")
with np.printoptions(precision=2, suppress=True, linewidth=95):
print(predictions*100)
# ---- Graphic visualisation
#
print("\nGraphically :\n")
plt.figure(figsize=(12,2))
plt.bar(range(43), predictions[0], align='center', alpha=0.5)
plt.ylabel('Probability')
plt.ylim((0,1))
plt.xlabel('Class')
plt.title('Trafic Sign prediction')
plt.show()
# ---- Predict class
#
p = np.argmax(predictions)
# ---- Show result
#
print("\nPrediction on the left, real stuff on the right :\n")
ooo.plot_images([x,x_meta[y]], [p,y], range(2), columns=3, x_size=3, y_size=2)
if p==y:
print("YEEES ! that's right!")
else:
print("oups, that's wrong ;-(")
```
%% Cell type:markdown id: tags:
---
That's all folks !
%% Cell type:code id: tags:
``` python
```
......
source diff could not be displayed: it is too large. Options to address this: view the blob.
%% Cell type:markdown id: tags:
German Traffic Sign Recognition Benchmark (GTSRB)
=================================================
---
Introduction au Deep Learning (IDLE) - S. Aria, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020
## Episode 3 : Tracking, visualizing and save models
Our main steps:
- Monitoring and understanding our model training
- Analyze the results
- Improving our model
- Add recovery points
## 1/ Import and init
%% Cell type:code id: tags:
``` python
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.callbacks import TensorBoard
import numpy as np
import h5py
import matplotlib.pyplot as plt
import os, time, random
import idle.pwk as ooo
from importlib import reload
ooo.init()
```
%% Output
IDLE 2020 - Practical Work Module
Version : 0.1.4
Run time : Monday 13 January 2020, 21:13:18
Matplotlib style : idle/talk.mplstyle
TensorFlow version : 2.0.0
Keras version : 2.2.4-tf
%% Cell type:markdown id: tags:
## 2/ Reload dataset
Dataset is one of the saved dataset: RGB25, RGB35, L25, L35, etc.
First of all, we're going to use a smart dataset : **set-24x24-L**
(with a GPU, it only takes 35'' compared to more than 5' with a CPU !)
%% Cell type:code id: tags:
``` python
%%time
dataset ='set-24x24-L'
# ---- Read dataset
#
filename='./data/'+dataset+'.h5'
with h5py.File(filename) as f:
x_train = f['x_train'][:]
y_train = f['y_train'][:]
x_test = f['x_test'][:]
y_test = f['y_test'][:]
# ---- Dataset shape
#
(n,lx,ly,lz) = x_train.shape
data_shape = (lx, ly, lz)
# ---- done
print('Dataset loaded ({:.1f} Mo)\n'.format(os.path.getsize(filename)/(1024*1024)))
```
%% Output
Dataset loaded (228.8 Mo)
CPU times: user 0 ns, sys: 297 ms, total: 297 ms
Wall time: 294 ms
%% Cell type:markdown id: tags:
## 3/ Have a look to the dataset
Note: Data must be reshape for matplotlib
%% Cell type:code id: tags:
``` python
print("x_train : ", x_train.shape)
print("y_train : ", y_train.shape)
print("x_test : ", x_test.shape)
print("y_test : ", y_test.shape)
ooo.plot_images(x_train, y_train, range(36), columns=12, x_size=1, y_size=1)
```
%% Output
x_train : (39209, 24, 24, 1)
y_train : (39209,)
x_test : (12630, 24, 24, 1)
y_test : (12630,)
%% Cell type:markdown id: tags:
## 4/ Create model
Some hyperparameters :
%% Cell type:code id: tags:
``` python
batch_size = 64
num_classes = 43
epochs = 8
```
%% Cell type:markdown id: tags:
My models :
%% Cell type:code id: tags:
``` python
def get_model_v1():
model = keras.models.Sequential()
model.add( keras.layers.Conv2D(96, (3,3), activation='relu', input_shape=(lx,ly,lz)))
model.add( keras.layers.MaxPooling2D((2, 2)))
model.add( keras.layers.Conv2D(192, (3, 3), activation='relu'))
model.add( keras.layers.MaxPooling2D((2, 2)))
model.add( keras.layers.Flatten())
model.add( keras.layers.Dense(500, activation='relu'))
model.add( keras.layers.Dense(500, activation='relu'))
model.add( keras.layers.Dense(43, activation='softmax'))
return model
```
%% Cell type:code id: tags:
``` python
# ---- The model I want to test..
#
model = get_model_v1()
model.summary()
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
```
%% Output
Model: "sequential_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_4 (Conv2D) (None, 22, 22, 96) 960
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 11, 11, 96) 0
_________________________________________________________________
conv2d_5 (Conv2D) (None, 9, 9, 192) 166080
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 4, 4, 192) 0
_________________________________________________________________
flatten_2 (Flatten) (None, 3072) 0
_________________________________________________________________
dense_6 (Dense) (None, 500) 1536500
_________________________________________________________________
dense_7 (Dense) (None, 500) 250500
_________________________________________________________________
dense_8 (Dense) (None, 43) 21543
=================================================================
Total params: 1,975,583
Trainable params: 1,975,583
Non-trainable params: 0
_________________________________________________________________
%% Cell type:markdown id: tags:
## 4/ Data augmentation
%% Cell type:code id: tags:
``` python
datagen = keras.preprocessing.image.ImageDataGenerator(featurewise_center=False,
featurewise_std_normalization=False,
width_shift_range=0.1,
height_shift_range=0.1,
zoom_range=0.2,
shear_range=0.1,
rotation_range=10.)
datagen.fit(x_train[:2000])
```
%% Cell type:markdown id: tags:
## 5/ Run model
%% Cell type:code id: tags:
``` python
%%time
# ---- Shuffle train data
# x_train,y_train=ooo.shuffle_np_dataset(x_train,y_train)
# ---- Train
# Note: To be faster in our example, we take only 2000 values
# but in the real world, we'd take the whole dataset!
#
history = model.fit(
x_train[:2000], y_train[:2000],
# datagen.flow(x_train[:2000], y_train[:2000], batch_size=batch_size),
# batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test[:200], y_test[:200]))
# model.save('./run/models/last-model.h5')
```
%% Output
Train on 2000 samples, validate on 200 samples
Epoch 1/8
2000/2000 [==============================] - 2s 767us/sample - loss: 0.2712 - accuracy: 0.9195 - val_loss: 1.0751 - val_accuracy: 0.7650
Epoch 2/8
2000/2000 [==============================] - 2s 829us/sample - loss: 0.2290 - accuracy: 0.9295 - val_loss: 1.1727 - val_accuracy: 0.7400
Epoch 3/8
2000/2000 [==============================] - 2s 906us/sample - loss: 0.1570 - accuracy: 0.9515 - val_loss: 1.0644 - val_accuracy: 0.8350
Epoch 4/8
2000/2000 [==============================] - 2s 896us/sample - loss: 0.1282 - accuracy: 0.9640 - val_loss: 1.0879 - val_accuracy: 0.8150
Epoch 5/8
2000/2000 [==============================] - 2s 913us/sample - loss: 0.0847 - accuracy: 0.9750 - val_loss: 1.1590 - val_accuracy: 0.8050
Epoch 6/8
2000/2000 [==============================] - 2s 920us/sample - loss: 0.0754 - accuracy: 0.9810 - val_loss: 1.2716 - val_accuracy: 0.8100
Epoch 7/8
2000/2000 [==============================] - 2s 930us/sample - loss: 0.0910 - accuracy: 0.9750 - val_loss: 1.1533 - val_accuracy: 0.8350
Epoch 8/8
2000/2000 [==============================] - 2s 973us/sample - loss: 0.0671 - accuracy: 0.9825 - val_loss: 1.1136 - val_accuracy: 0.8350
CPU times: user 1min 21s, sys: 7.22 s, total: 1min 29s
Wall time: 14.3 s
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
Running Tensorboard from Jupyter lab
====================================
---
Introduction au Deep Learning (IDLE) - S. Arias, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020
Vesion : 1.0
%% Cell type:markdown id: tags:
## Méthode 1 : Shell execute
**Start**
%% Cell type:code id: tags:
``` python
%%bash
tensorboard_start --logdir ./run/logs
```
%% Output
Tensorbord started with pid 84593
Tensorbord started with pid 80451
%% Cell type:code id: tags:
``` python
%%bash
tensorboard_stop
```
%% Output
Tensorboard process not found...
%% Cell type:code id: tags:
``` python
%%bash
# ---- Port number,
PORT_JPY="$(id -u)"
PORT_TSB="$(( $PORT_J + 10000 ))"
HOST_G="$(hostname)"
SSH_CMD="/usr/bin/ssh -NL 8888:$HOST_G:$PORT_J -L 6006:$HOST_G:$PORT_T dahu.ciment"
tensorboard --port 21277 --host 0.0.0.0 --logdir ./run/logs &>/dev/null &
# tensorboard --port 21277 --host 0.0.0.0 --logdir ./run/logs &>/dev/null &
tensorboard --port 21277 --host 0.0.0.0 --logdir ./run/logs 2>&1
```
%% Output
W0113 22:02:28.498013 140212267140864 plugin_event_accumulator.py:294] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
TensorBoard 2.0.0 at http://0.0.0.0:21277/ (Press CTRL+C to quit)
Traceback (most recent call last):
File "/home/paroutyj/.conda/envs/deeplearning2/bin/tensorboard", line 10, in <module>
sys.exit(run_main())
File "/home/paroutyj/.conda/envs/deeplearning2/lib/python3.7/site-packages/tensorboard/main.py", line 64, in run_main
app.run(tensorboard.main, flags_parser=tensorboard.configure)
File "/home/paroutyj/.conda/envs/deeplearning2/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/paroutyj/.conda/envs/deeplearning2/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "/home/paroutyj/.conda/envs/deeplearning2/lib/python3.7/site-packages/tensorboard/program.py", line 222, in main
self._register_info(server)
File "/home/paroutyj/.conda/envs/deeplearning2/lib/python3.7/site-packages/tensorboard/program.py", line 268, in _register_info
manager.write_info_file(info)
File "/home/paroutyj/.conda/envs/deeplearning2/lib/python3.7/site-packages/tensorboard/manager.py", line 268, in write_info_file
with open(_get_info_file_path(), "w") as outfile:
PermissionError: [Errno 13] Permission denied: '/tmp/.tensorboard-info/pid-94212.info'
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
<ipython-input-19-d5c18c938787> in <module>
----> 1 get_ipython().run_cell_magic('bash', '', '\n# ---- Port number, \nPORT_JPY="$(id -u)"\nPORT_TSB="$(( $PORT_J + 10000 ))"\nHOST_G="$(hostname)"\nSSH_CMD="/usr/bin/ssh -NL 8888:$HOST_G:$PORT_J -L 6006:$HOST_G:$PORT_T dahu.ciment"\n\n# tensorboard --port 21277 --host 0.0.0.0 --logdir ./run/logs &>/dev/null &\ntensorboard --port 21277 --host 0.0.0.0 --logdir ./run/logs 2>&1\n')
~/.conda/envs/deeplearning2/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2350 with self.builtin_trap:
2351 args = (magic_arg_s, cell)
-> 2352 result = fn(*args, **kwargs)
2353 return result
2354
~/.conda/envs/deeplearning2/lib/python3.7/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
140 else:
141 line = script
--> 142 return self.shebang(line, cell)
143
144 # write a basic docstring:
</home/paroutyj/.conda/envs/deeplearning2/lib/python3.7/site-packages/decorator.py:decorator-gen-110> in shebang(self, line, cell)
~/.conda/envs/deeplearning2/lib/python3.7/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
185 # but it's overkill for just that one bit of state.
186 def magic_deco(arg):
--> 187 call = lambda f, *a, **k: f(*a, **k)
188
189 if callable(arg):
~/.conda/envs/deeplearning2/lib/python3.7/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
243 sys.stderr.flush()
244 if args.raise_error and p.returncode!=0:
--> 245 raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
246
247 def _run_script(self, p, cell, to_close):
CalledProcessError: Command 'b'\n# ---- Port number, \nPORT_JPY="$(id -u)"\nPORT_TSB="$(( $PORT_J + 10000 ))"\nHOST_G="$(hostname)"\nSSH_CMD="/usr/bin/ssh -NL 8888:$HOST_G:$PORT_J -L 6006:$HOST_G:$PORT_T dahu.ciment"\n\n# tensorboard --port 21277 --host 0.0.0.0 --logdir ./run/logs &>/dev/null &\ntensorboard --port 21277 --host 0.0.0.0 --logdir ./run/logs 2>&1\n'' returned non-zero exit status 1.
%% Cell type:code id: tags:
``` python
!echo "This a test">/tmp/.tensorboard-info/foobar-655465.txt
#!cat /tmp/foobar-655465.txt
#!rm /tmp/foobar-655465.txt
!ls -ld /tmp/.tensorboard-info
!ls -ld /tmp/.tensorboard-info/*
See : https://github.com/tensorflow/tensorboard/issues/2010
```
%% Output
/bin/sh: 1: cannot create /tmp/.tensorboard-info/foobar-655465.txt: Permission denied
drwxr-xr-x 2 juanbaldonado l-inria 60 Jan 13 21:00 /tmp/.tensorboard-info
-rw-r--r-- 1 juanbaldonado l-inria 346 Jan 13 21:00 /tmp/.tensorboard-info/pid-74153.info
%% Cell type:markdown id: tags:
**Check**
%% Cell type:code id: tags:
``` python
!echo $(ps ax | grep 'tensorboard --port 21277' | grep -v grep | awk '{print $1}')
```
%% Output
84593
%% Cell type:code id: tags:
``` python
!ps ax | grep tensorboard
!pwd
```
%% Output
83294 pts/0 Ss+ 0:00 /bin/sh -c ps ax | grep tensorboard
83296 pts/0 S+ 0:00 grep tensorboard
/home/paroutyj/fidle/GTSRB
%% Cell type:markdown id: tags:
**Stop**
%% Cell type:code id: tags:
``` python
%%bash
p="$(ps ax | grep 'tensorboard --port 21277' | grep -v grep | awk '{print $1}')"
if [ -z ${p} ]; then echo "No process"; else kill $p; echo "Ended ($p)" ; fi
```
%% Output
Ended (82555)
%% Cell type:markdown id: tags:
## Méthode 2 : Magic command
**Start**
%% Cell type:code id: tags:
``` python
%load_ext tensorboard
```
%% Cell type:code id: tags:
``` python
%tensorboard --port 21277 --host 0.0.0.0 --logdir ./run/logs
```
%% Cell type:markdown id: tags:
**Stop**
No way... use bash method
## Methode 3 : Tensorboard module
**Start**
%% Cell type:code id: tags:
``` python
import tensorboard.notebook as tsb
```
%% Cell type:code id: tags:
``` python
tsb.start('--port 21277 --host 0.0.0.0 --logdir ./run/logs')
```
%% Output
%% Cell type:markdown id: tags:
**Check**
%% Cell type:code id: tags:
``` python
a=tsb.list()
```
%% Cell type:markdown id: tags:
**Stop**
No way... use bash method
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment