Project Summary

The first ever successful numerical weather prediction (NWP) was done on the first ever electronic computer, the ENIAC. Since then, NWP has continued to push the envelope of high-performance computing. For example, to cover the global atmosphere at 1 km horizontal resolution necessary to resolve active thunderstorms, approximately 50 billion grid points are needed. To produce global ensemble forecasts at this resolution, computers capable of tens to hundreds of petaflops sustained will be required.

The currently installed and emerging NSF Track 2 supercomputing systems, capable of up to 1 petaflops peak performance, and the planned installation at NCSA by 2011 of a leadership class Track 1 system capable of 1 petaflops sustained, offer an unprecedented opportunity for transformative scientific and technological advances, and allow for NWP research and experimentations that pave the way for future operational implementations. NWP and their data assimilation systems will, however, have to be adapted, and in some cases substantially redesigned, to take maximal advantage of the multi-level storage architectures of multicore and many-core processors, and to overcome limitations associated with ongoing performance divergence among the various elements of the storage hierarchy. Such problems associated with data movement across the memory hierarchy become more acute as CPUs consist of ever more cores, and as the system is comprised of ever more networked nodes, and as these growing numbers of nodes access shared filesystems comprising ever more components.

In the project, a team of world leading atmospheric and computer scientists and weather prediction system developers is formed to tackle the most challenging problems of (a) very-high-resolution NWP, (b) obtaining the optimal state estimations for initializing ensembles of predictions by assimilating the highest volume of weather observations available, and (c) addressing problem sizes and scales that are only attainable on petascale computing platforms.

Intellectual Merit An ensemble-based scalable petascale data assimilation system will be developed. The system will adopt a much more scalable local ensemble transform Kalman filter (LETKF) algorithm, and will seek to achieve order of magnitude performance gain for 4D high-resolution high-frequency ensemble Kalman filtering, by keeping all prediction and data assimilation components and some of the data and post-processing in the fastest levels of the storage hierarchy possible (for example, the caches and main memory), and by avoiding the biggest bottleneck of disk I/O associated with such extremely data intensive applications. A hybrid distributed plus shared memory parallel model will be adopted for portions of the system, to minimize data movement and to help improve load balance. General communication APIs and toolkits will be developed/adopted to facilitate the large amount of data movement among nodes and cores, and communication/computation overlap will be exploited whenever possible. Runtime management and dynamic load balancing will be built to minimize resource waste. Lower level optimizations that improve, for example, multicore utilization and cache reuse, will be achieved via performance and code analysis and profiling. By making freely available the developed systems, this PetaApps team will enable the research community to carry out continental-scale ensemble data assimilation and prediction for severe thunderstorms, and to apply the system to, for example, the prediction and predictability studies of hurricanes and tornadoes that otherwise would not be possible. The expected improvement to severe weather prediction will provide the capability to significantly mitigate the negative impacts of weather on the economy and human lives. Furthermore, the strategies developed here can be applied to the prediction of other geophysical and dynamical systems.

Broader Impact The project will provide hands-on training to graduate students and post-docs, and involve undergraduates in an interdisciplinary environment, on topics including advanced data assimilation, dynamics and predictability of high-impact weather, high performance computing, algorithm and compiler optimizations, and computer hardware design and architectures, and will help to train a new generation of engineers and scientists who can lead our nation in taking maximum advantage of what these emerging computing technologies have to offer. Its research results will be incorporated into the curriculum of Meteorology, Computer Science, and HPC training. Special efforts will be made to recruit and involve under-represented groups, in particular, Native American and Hispanic. The software research to be conducted has the potential to result in new optimization algorithms, compiler improvements, improved data assimilation algorithms and techniques, and data assimilation capabilities that enable transformative scientific investigations and discoveries.