In the last years, Neural Networks have enriched the state-of-the-art in probabilistic modeling. This is principally due to the advances in deep learning which allow a better understanding of complex systems. However, the stochastic representation of spatio-temporal fields is still an open challenge that may benefit from the recent advances in probabilistic modelization. In this work, we explore neural network to derive a stochastic representation of spatio-temporal dynamical systems based on ensemble forecasting. Trough the implementation of our stochastic model in a classical Kalman filtering scheme, we demonstrate the relevance of the proposed architecture in the reconstruction of geophysical fields with respect to the state-of-the-art approaches.