Skip to content

Commit e14ffdc

Browse files
cehbrechtcehbrecht
authored
Demo c4i esgf (#54)
* added c4i example * add pymetalink * update cmip6 testdata * updated monitor notebooks * disable nb monitor checks in ci
1 parent da65eb0 commit e14ffdc

10 files changed

+1209
-50
lines changed

‎.github/workflows/main.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ jobs:
3030
- name: Test with pytest
3131
run: |
3232
pytest -v tests
33-
- name: Test monitor notebooks with pytest
34-
run: |
35-
pytest -v --nbval notebooks/monitor
36-
if: matrix.python-version == 3.6
33+
# - name: Test monitor notebooks with pytest
34+
# run: |
35+
# pytest -v --nbval notebooks/monitor
36+
# if: matrix.python-version == 3.6

‎environment.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ dependencies:
88
#- openssl
99
- birdy
1010
- beautifulsoup4
11+
- pymetalink
1112
# open xarray results
1213
- netcdf4
1314
- xarray>=0.16

‎notebooks/c4i/c4i-access-cmip6-subset-average.ipynb

Lines changed: 372 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,372 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "frank-sense",
6+
"metadata": {},
7+
"source": [
8+
"# Demo with esgf search for CMIP6 at DKRZ site (Subset + Average)\n",
9+
"\n",
10+
"ESGF Node at DKRZ: https://esgf-data.dkrz.de/search/cmip6-dkrz/"
11+
]
12+
},
13+
{
14+
"cell_type": "markdown",
15+
"id": "blond-english",
16+
"metadata": {},
17+
"source": [
18+
"## Use esgf search at DKRZ ... no distributed search\n",
19+
"\n",
20+
"\n",
21+
"\n",
22+
"Using ``esgf-pyclient``: \n",
23+
"https://esgf-pyclient.readthedocs.io/en/latest/notebooks/examples/search.html"
24+
]
25+
},
26+
{
27+
"cell_type": "code",
28+
"execution_count": null,
29+
"id": "incorporate-wings",
30+
"metadata": {},
31+
"outputs": [],
32+
"source": [
33+
"from pyesgf.search import SearchConnection\n",
34+
"conn = SearchConnection('http://esgf-data.dkrz.de/esg-search',\n",
35+
" distrib=False)"
36+
]
37+
},
38+
{
39+
"cell_type": "markdown",
40+
"id": "complex-teacher",
41+
"metadata": {},
42+
"source": [
43+
"**Search only CMIP6 files locally available at DKRZ**"
44+
]
45+
},
46+
{
47+
"cell_type": "code",
48+
"execution_count": null,
49+
"id": "civilian-monaco",
50+
"metadata": {},
51+
"outputs": [],
52+
"source": [
53+
"ctx = conn.new_context(project='CMIP6', data_node='esgf3.dkrz.de', latest=True, replica=False)\n",
54+
"ctx.hit_count"
55+
]
56+
},
57+
{
58+
"cell_type": "markdown",
59+
"id": "three-thomson",
60+
"metadata": {},
61+
"source": [
62+
"Select only one dataset"
63+
]
64+
},
65+
{
66+
"cell_type": "code",
67+
"execution_count": null,
68+
"id": "color-courtesy",
69+
"metadata": {},
70+
"outputs": [],
71+
"source": [
72+
"results = ctx.search(\n",
73+
" institution_id='MPI-M',\n",
74+
" source_id='MPI-ESM1-2-HR',\n",
75+
" experiment_id='historical', \n",
76+
" variable='tas', \n",
77+
" frequency='day',\n",
78+
" variant_label='r1i1p1f1'\n",
79+
")\n",
80+
"len(results)"
81+
]
82+
},
83+
{
84+
"cell_type": "code",
85+
"execution_count": null,
86+
"id": "satisfactory-details",
87+
"metadata": {},
88+
"outputs": [],
89+
"source": [
90+
"ds = results[0]\n",
91+
"ds.json"
92+
]
93+
},
94+
{
95+
"cell_type": "markdown",
96+
"id": "oriented-collectible",
97+
"metadata": {},
98+
"source": [
99+
"Get a dataset identifier used by rook"
100+
]
101+
},
102+
{
103+
"cell_type": "code",
104+
"execution_count": null,
105+
"id": "expanded-validation",
106+
"metadata": {},
107+
"outputs": [],
108+
"source": [
109+
"dataset_id = ds.json['instance_id']\n",
110+
"dataset_id"
111+
]
112+
},
113+
{
114+
"cell_type": "markdown",
115+
"id": "royal-tourism",
116+
"metadata": {},
117+
"source": [
118+
"Time range"
119+
]
120+
},
121+
{
122+
"cell_type": "code",
123+
"execution_count": null,
124+
"id": "intellectual-medium",
125+
"metadata": {},
126+
"outputs": [],
127+
"source": [
128+
"f\"{ds.json['datetime_start']}/{ds.json['datetime_stop']})\""
129+
]
130+
},
131+
{
132+
"cell_type": "markdown",
133+
"id": "frozen-bowling",
134+
"metadata": {},
135+
"source": [
136+
"Bounding Box: (West, Sout, East, North)"
137+
]
138+
},
139+
{
140+
"cell_type": "code",
141+
"execution_count": null,
142+
"id": "secure-clark",
143+
"metadata": {},
144+
"outputs": [],
145+
"source": [
146+
"f\"({ds.json['west_degrees']}, {ds.json['south_degrees']},{ds.json['east_degrees']}, {ds.json['west_degrees']}, {ds.json['north_degrees']})\"\n"
147+
]
148+
},
149+
{
150+
"cell_type": "markdown",
151+
"id": "presidential-mixer",
152+
"metadata": {},
153+
"source": [
154+
"Size in GB"
155+
]
156+
},
157+
{
158+
"cell_type": "code",
159+
"execution_count": null,
160+
"id": "controlling-benchmark",
161+
"metadata": {},
162+
"outputs": [],
163+
"source": [
164+
"f\"{ds.json['size'] / 1024 / 1024 / 1024} GB\""
165+
]
166+
},
167+
{
168+
"cell_type": "markdown",
169+
"id": "working-pulse",
170+
"metadata": {},
171+
"source": [
172+
"## Use Rook to run subset + average"
173+
]
174+
},
175+
{
176+
"cell_type": "code",
177+
"execution_count": null,
178+
"id": "deadly-environment",
179+
"metadata": {},
180+
"outputs": [],
181+
"source": [
182+
"import os\n",
183+
"os.environ['ROOK_URL'] = 'http://rook1.cloud.dkrz.de/wps'\n",
184+
"os.environ['ROOK_MODE'] = 'async'\n",
185+
"\n",
186+
"from rooki import operators as ops"
187+
]
188+
},
189+
{
190+
"cell_type": "markdown",
191+
"id": "stylish-comedy",
192+
"metadata": {},
193+
"source": [
194+
"Run subset workflow\n",
195+
"\n",
196+
"http://bboxfinder.com/"
197+
]
198+
},
199+
{
200+
"cell_type": "code",
201+
"execution_count": null,
202+
"id": "characteristic-elizabeth",
203+
"metadata": {},
204+
"outputs": [],
205+
"source": [
206+
"bbox_africa = \"-23.906250,-35.746512,63.632813,37.996163\"\n",
207+
"\n",
208+
"\n",
209+
"subset = ops.Subset(\n",
210+
" ops.Input(\n",
211+
" 'tas', [dataset_id]\n",
212+
" ),\n",
213+
" time=\"1850-01-01/1850-12-31\",\n",
214+
" area=bbox_africa,\n",
215+
" )\n",
216+
"wf = ops.Average(subset, dims=\"time\")\n",
217+
"\n",
218+
"resp = wf.orchestrate()\n",
219+
"resp.ok"
220+
]
221+
},
222+
{
223+
"cell_type": "markdown",
224+
"id": "selective-cylinder",
225+
"metadata": {},
226+
"source": [
227+
"Metalink URL"
228+
]
229+
},
230+
{
231+
"cell_type": "code",
232+
"execution_count": null,
233+
"id": "acquired-cable",
234+
"metadata": {},
235+
"outputs": [],
236+
"source": [
237+
"resp.url"
238+
]
239+
},
240+
{
241+
"cell_type": "markdown",
242+
"id": "hollow-edmonton",
243+
"metadata": {},
244+
"source": [
245+
"Number of files"
246+
]
247+
},
248+
{
249+
"cell_type": "code",
250+
"execution_count": null,
251+
"id": "little-copyright",
252+
"metadata": {},
253+
"outputs": [],
254+
"source": [
255+
"resp.num_files"
256+
]
257+
},
258+
{
259+
"cell_type": "markdown",
260+
"id": "different-beatles",
261+
"metadata": {},
262+
"source": [
263+
"Total size in MB"
264+
]
265+
},
266+
{
267+
"cell_type": "code",
268+
"execution_count": null,
269+
"id": "mighty-attitude",
270+
"metadata": {},
271+
"outputs": [],
272+
"source": [
273+
"resp.size_in_mb"
274+
]
275+
},
276+
{
277+
"cell_type": "markdown",
278+
"id": "registered-karaoke",
279+
"metadata": {},
280+
"source": [
281+
"Download URLs"
282+
]
283+
},
284+
{
285+
"cell_type": "code",
286+
"execution_count": null,
287+
"id": "driving-theory",
288+
"metadata": {},
289+
"outputs": [],
290+
"source": [
291+
"resp.download_urls()"
292+
]
293+
},
294+
{
295+
"cell_type": "markdown",
296+
"id": "reasonable-vacation",
297+
"metadata": {},
298+
"source": [
299+
"Download and open with xarray"
300+
]
301+
},
302+
{
303+
"cell_type": "code",
304+
"execution_count": null,
305+
"id": "collectible-enough",
306+
"metadata": {},
307+
"outputs": [],
308+
"source": [
309+
"ds_0 = resp.datasets()[0]\n",
310+
"ds_0"
311+
]
312+
},
313+
{
314+
"cell_type": "markdown",
315+
"id": "manufactured-judgment",
316+
"metadata": {},
317+
"source": [
318+
"Provenance: URL to json document"
319+
]
320+
},
321+
{
322+
"cell_type": "code",
323+
"execution_count": null,
324+
"id": "patient-italy",
325+
"metadata": {},
326+
"outputs": [],
327+
"source": [
328+
"resp.provenance()"
329+
]
330+
},
331+
{
332+
"cell_type": "markdown",
333+
"id": "liable-today",
334+
"metadata": {},
335+
"source": [
336+
"Provenance Plot"
337+
]
338+
},
339+
{
340+
"cell_type": "code",
341+
"execution_count": null,
342+
"id": "dimensional-romance",
343+
"metadata": {},
344+
"outputs": [],
345+
"source": [
346+
"from IPython.display import Image\n",
347+
"Image(resp.provenance_image())"
348+
]
349+
}
350+
],
351+
"metadata": {
352+
"kernelspec": {
353+
"display_name": "Python 3",
354+
"language": "python",
355+
"name": "python3"
356+
},
357+
"language_info": {
358+
"codemirror_mode": {
359+
"name": "ipython",
360+
"version": 3
361+
},
362+
"file_extension": ".py",
363+
"mimetype": "text/x-python",
364+
"name": "python",
365+
"nbconvert_exporter": "python",
366+
"pygments_lexer": "ipython3",
367+
"version": "3.9.2"
368+
}
369+
},
370+
"nbformat": 4,
371+
"nbformat_minor": 5
372+
}

0 commit comments

Comments
 (0)