pres09_practical1.ipynb 7.71 KB
Newer Older
paugier's avatar
paugier committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Python training UGA 2017\n",
    "\n",
    "**A training to acquire strong basis in Python to use it efficiently**\n",
    "\n",
Loic Huder's avatar
Loic Huder committed
15
    "Pierre Augier (LEGI), Cyrille Bonamy (LEGI), Eric Maldonado (Irstea), Franck Thollard (ISTerre), Christophe Picard (LJK), Loïc Huder (ISTerre)\n",
paugier's avatar
paugier committed
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    "\n",
    "## Practical session 1\n",
    "\n",
    "File parsing and dictionary usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Goal\n",
    "\n",
Eric Maldonado's avatar
Eric Maldonado committed
32
    "The goal of this session is to end up with a script that computes some simple statistics from Meteo open data files. The file was modified and reduced for this exercice (just 1 station with data in just one year : 2016). In future, you can download other data here : https://public.opendatasoft.com/explore/dataset/donnees-synop-essentielles-omm/export/\n",
paugier's avatar
paugier committed
33
34
35
    "\n",
    "## Material \n",
    "\n",
Maldonado Eric's avatar
Maldonado Eric committed
36
    "The file contains lines of the form:\n",
paugier's avatar
paugier committed
37
    "\n",
Maldonado Eric's avatar
Maldonado Eric committed
38
    "ID OMM station,Date,Average wind 10 mn,Temperature,Humidity,Rainfall 3 last hours,Station\n",
paugier's avatar
paugier committed
39
    "\n",
Maldonado Eric's avatar
Maldonado Eric committed
40
41
42
43
44
    "7761,2016-01-01T01:00:00+01:00,2.0,283.75,94,0.2,AJACCIO\n",
    "7761,2016-01-01T04:00:00+01:00,2.2,283.95,91,0.2,AJACCIO\n",
    "7761,2016-01-01T07:00:00+01:00,1.7,284.05,88,0.2,AJACCIO\n",
    "7761,2016-01-01T10:00:00+01:00,1.6,287.05,75,0.2,AJACCIO\n",
    "7761,2016-01-01T13:00:00+01:00,3.1,289.55,73,0.0,AJACCIO\n",
paugier's avatar
paugier committed
45
    "\n",
Maldonado Eric's avatar
Maldonado Eric committed
46
47
    "This is a classic csv file with separated data by  \",\"\n",
    "The first line is the header.\n",
paugier's avatar
paugier committed
48
    "\n",
Maldonado Eric's avatar
Maldonado Eric committed
49
    "## Information to extract\n",
paugier's avatar
paugier committed
50
    "\n",
Eric Maldonado's avatar
Eric Maldonado committed
51
    "We want to compute, some statistics for this station \n",
paugier's avatar
paugier committed
52
    "\n",
Maldonado Eric's avatar
Maldonado Eric committed
53
    "## Warning\n",
paugier's avatar
paugier committed
54
    "\n",
paugier's avatar
paugier committed
55
    "The temperature measurement is in kelvin (273,15 K $\\leftrightarrow$ 0 °C)\n"
paugier's avatar
paugier committed
56
57
58
59
60
61
62
63
64
65
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
Loic Huder's avatar
Loic Huder committed
66
    "# Step 1: load data\n",
Maldonado Eric's avatar
Maldonado Eric committed
67
    "\n",
paugier's avatar
paugier committed
68
    "Write a script with a function `load_data()` that\n",
Maldonado Eric's avatar
Maldonado Eric committed
69
    "  - open the file\n",
Loic Huder's avatar
Loic Huder committed
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
    "  - load data in one of the following structures (more details below):\n",
    "      - 1.1 Single dictionnary\n",
    "      - 1.2 Multiple structures\n",
    "      - 1.3 Class instance (object-oriented)\n",
    "\n",
    "## 1.1: Single dictionnary (pres07)\n",
    "Load in the data in a single dictionnary of this structure:\n",
    "```python\n",
    "{'Date': [wind,temperature,humidity,rainfall]}\n",
    "```\n",
    "\n",
    "For example\n",
    "```python\n",
    "{\n",
    "    '2016-01-01T01': [2.0,283.75,94,0.2],\n",
    "    '2016-01-01T04': [2.2,283.95,91,0.2]\n",
    "}\n",
    "```\n",
Maldonado Eric's avatar
Maldonado Eric committed
88
    "      \n",
paugier's avatar
paugier committed
89
90
91
92
    "In this case, we can consider YYYY-MM-DDTHH as the key for the station dictionary.\n",
    "\n",
    "Split each line and extract data.\n",
    "\n",
Loic Huder's avatar
Loic Huder committed
93
    "#### Hint\n",
Maldonado Eric's avatar
Maldonado Eric committed
94
    "You can use the method *split* from the str class.\n"
paugier's avatar
paugier committed
95
96
97
98
   ]
  },
  {
   "cell_type": "code",
paugier's avatar
paugier committed
99
   "execution_count": 1,
paugier's avatar
paugier committed
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', 'am', 'lucky']\n"
     ]
    }
   ],
   "source": [
    "s = \"I am lucky\"\n",
    "l = s.split()\n",
    "print(l)"
   ]
  },
Loic Huder's avatar
Loic Huder committed
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2: Multiple structures (pres07)\n",
    "Load the data in multiple dictionnaries or lists (one per field).\n",
    "#### Example for dictionnaries:\n",
    "You can use the following structure\n",
    "```python3\n",
    "wind = {'Date1': wind_value1, 'Date2: wind_value2, ...}\n",
    "temperature = {'Date1': temperature1, 'Date2: temperature2, ...}\n",
    "...\n",
    "```\n",
    "#### Example for lists:\n",
    "You can use the following structure\n",
    "```python3\n",
    "dates = ['Date1', 'Date2', ...]\n",
    "wind = [wind_value1, wind_value2, ...]\n",
    "temperature = [temperature1, temperature2, ...]\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3: Class instance (pres08)\n",
    "Load the data in an instance of a class `WeatherStation` that you will define yourself. `load_data()` can therefore be a method of this class.\n",
    "\n",
    "#### Hint :\n",
    "This is very similar as 1.2. The only difference is that the structures storing the data are attributes of a class."
   ]
  },
paugier's avatar
paugier committed
154
155
156
157
158
159
160
161
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
Loic Huder's avatar
Loic Huder committed
162
    "# Step 2: Compute max temperature and average temperature for the station\n",
paugier's avatar
paugier committed
163
    "\n",
Eric Maldonado's avatar
Eric Maldonado committed
164
    "Write 2 functions `get_max_temperature()` and `get_average_temperature()` that:\n",
paugier's avatar
paugier committed
165
    "\n",
Maldonado Eric's avatar
Maldonado Eric committed
166
167
    "- return a float \n",
    "\n"
paugier's avatar
paugier committed
168
169
170
171
172
173
174
175
176
177
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
Loic Huder's avatar
Loic Huder committed
178
    "# Step 3: Compute sum of the rainfall for one station\n",
Maldonado Eric's avatar
Maldonado Eric committed
179
    "\n",
Eric Maldonado's avatar
Eric Maldonado committed
180
181
182
    "Write 1 function `get_sum_rainfall()` that sum the rainfall.\n",
    "\n",
    "- return a float\n",
paugier's avatar
paugier committed
183
    "\n",
paugier's avatar
paugier committed
184
    "Be careful, some measurement have no rainfall data.\n"
paugier's avatar
paugier committed
185
186
187
   ]
  },
  {
Maldonado Eric's avatar
Maldonado Eric committed
188
   "cell_type": "markdown",
paugier's avatar
paugier committed
189
   "metadata": {
Maldonado Eric's avatar
Maldonado Eric committed
190
191
192
    "slideshow": {
     "slide_type": "slide"
    }
paugier's avatar
paugier committed
193
194
   },
   "source": [
Loic Huder's avatar
Loic Huder committed
195
    "# Step 4: Search max period without rainfall\n",
paugier's avatar
paugier committed
196
    "\n",
Eric Maldonado's avatar
Eric Maldonado committed
197
    "Write 1 function `period_without_rainfall()`\n",
paugier's avatar
paugier committed
198
    "\n",
Maldonado Eric's avatar
Maldonado Eric committed
199
    "- return the beginning date, the ending date and the number of days without rainfall\n",
paugier's avatar
paugier committed
200
    "\n",
paugier's avatar
paugier committed
201
    "## Hint\n",
Loic Huder's avatar
Loic Huder committed
202
    "This is the syntax to return multiple values in a function:\n",
paugier's avatar
paugier committed
203
204
205
206
    "\n",
    "```\n",
    "return date_min, date_max, period_max / 8\n",
    "```"
paugier's avatar
paugier committed
207
208
209
210
211
212
213
214
215
216
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
Loic Huder's avatar
Loic Huder committed
217
    "# Step 5: How many hours with humidity rate < 60\n",
paugier's avatar
paugier committed
218
    "\n",
Eric Maldonado's avatar
Eric Maldonado committed
219
    "Write 1 function `get_hours_humidity(rate)`\n",
paugier's avatar
paugier committed
220
    "\n",
Eric Maldonado's avatar
Eric Maldonado committed
221
    "- takes 1 parameter :  the humidity rate \n",
paugier's avatar
paugier committed
222
    "- returns the number of days\n"
paugier's avatar
paugier committed
223
224
225
   ]
  },
  {
paugier's avatar
paugier committed
226
   "cell_type": "markdown",
paugier's avatar
paugier committed
227
   "metadata": {
Maldonado Eric's avatar
Maldonado Eric committed
228
229
230
    "slideshow": {
     "slide_type": "slide"
    }
paugier's avatar
paugier committed
231
   },
paugier's avatar
paugier committed
232
233
234
235
236
237
238
239
   "source": [
    "## Final remark: Pandas\n",
    "\n",
    "To do such data analysis, ones should not use pure Python code without external library! The library Pandas has been written to do this in few lines:"
   ]
  },
  {
   "cell_type": "code",
Loic Huder's avatar
Loic Huder committed
240
   "execution_count": 3,
paugier's avatar
paugier committed
241
   "metadata": {},
paugier's avatar
paugier committed
242
243
244
245
246
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
paugier's avatar
paugier committed
247
      "The average of temperature at Ajaccio is 16.3 °C.\n"
paugier's avatar
paugier committed
248
249
250
251
     ]
    }
   ],
   "source": [
Maldonado Eric's avatar
Maldonado Eric committed
252
    "import pandas\n",
paugier's avatar
paugier committed
253
    "\n",
Loic Huder's avatar
Loic Huder committed
254
    "df = pandas.read_csv(\n",
paugier's avatar
paugier committed
255
    "    '../TP/TP1_MeteoData/data/synop-2016.csv', sep=',', header=0)\n",
paugier's avatar
paugier committed
256
    "\n",
Maldonado Eric's avatar
Maldonado Eric committed
257
258
259
    "# print(df.columns)\n",
    "# print(df.age)\n",
    "# print(df['Temperature'])\n",
paugier's avatar
paugier committed
260
261
262
263
264
    "# print(df[(df['Station']=='AJACCIO')])\n",
    "# print(df[(df['Station'] == 'AJACCIO')]['Rainfall 3 last hours'].sum())\n",
    "\n",
    "temp = df[(df['Station'] == 'AJACCIO')]['Temperature'].mean()-273.15\n",
    "print(f'The average of temperature at Ajaccio is {temp:.1f} °C.')"
paugier's avatar
paugier committed
265
   ]
Eric Maldonado's avatar
Eric Maldonado committed
266
267
268
269
270
271
272
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
paugier's avatar
paugier committed
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
Loic Huder's avatar
Loic Huder committed
292
   "version": "3.6.7"
paugier's avatar
paugier committed
293
294
295
296
297
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}