The Pangeo Big Data Ecosystem and its use at CNES
| Type | Proceedings paper | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Date | 2019 | ||||||||
| Language | English | ||||||||
| Other localization | http://publications.jrc.ec.europa.eu/repository/handle/JRC115761 | ||||||||
| Author(s) | Eynard-Bontemps Guillaume1, Abernathey Ryan2, Hamman Joseph3, Ponte Aurelien 4, Rath Willi5 |
||||||||
| Affiliation(s) | 1 : Centre National d’Etudes Spatiales (CNES), Toulouse, France 2 : Columbia University / Lamont Doherty Earth Observatory, New-York, USA 3 : National Center for Atmospheric Research (NCAR), Boulder, USA 4 : Ifremer, Univ. Brest, CNRS, IRD, Laboratoire dOcanographie Physique et Spatiale (LOPS), IUEM, Brest 29280, France 5 : GEOMAR Helmholtz Centre for Ocean Research Kiel, Kiel, Germany |
||||||||
| Meeting | Big Data from Space (BiDS'19) .... Turning Data into insights... 19-21 fébruary 2019, Munich, Germany | ||||||||
| Source | P. Soille, S. Loekken, and S. Albani (Eds.) Proc. of the 2019 conference on Big Data from Space (BiDS’2019), EUR 29660 EN, Publications Office of the European Union, Luxembourg, 2019, ISBN 978-92-76-00034-1 , doi:10.2760/848593, 2019. Part. Interactive Processing and Visualisation, pp.49-52 | ||||||||
| Note | EUR 29660 EN OP KJ-NA-29660-EN-N (online) ISSN: 1831-9424 | ||||||||
| Keyword(s) | Pangeo, Dask, Jupyter, HPC, Cloud, Big Data, Analysis, Open Source | ||||||||
| Abstract | Pangeo[1] is a community-driven effort for open-source big data initially focused on the Earth System Sciences. One of its primary goals is to enable scientists in analyzing petascale datasets both on classical high-performance computing (HPC) and on public cloud infrastructure. In only a few years, Pangeo has grown into a very productive community collaborating on the development of open-source analysis tools for science. It provides a set of example deployments based on open-source Scientific Python packages like Jupyter[2], Dask[3], and Xarray[4] that bring together scientists and developer with their actual use-cases. In this paper, we first describe Pangeo ecosystem and community. We then present its impact on the work of scientists from CNES on the HPC deployment there. We conclude with a future outlook for Pangeo in this agency and beyond. |
||||||||
| Full Text |
|