The need to scale next-generation industrial engineering problems to the largest computational platforms presents unique challenges. This paper focuses on data management related problems faced by the Uintah simulation framework at a production scale of 260K processes. Uintah provides a highly scalable asynchronous many-task runtime system, which in this work is used for the modeling of a 1000 megawatt electric (MWe) ultra-supercritical (USC) coal boiler. At 260K processes, we faced both parallel I/O and visualization related challenges, e.g., the default file-per-process I/O approach of Uintah did not scale on Mira. In this paper we present a simple to implement, restructuring based parallel I/O technique. We impose a restructuring step that alters the distribution of data among processes. The goal is to distribute the dataset such that each process holds a larger chunk of data, which is then written to a file independently. This approach finds a middle ground between two of the most common parallel I/O schemes--file per process I/O and shared file I/O--in terms of both the total number of generated files, and the extent of communication involved during the data aggregation phase. To address scalability issues when visualizing the simulation data, we developed a lightweight renderer using OSPRay, which allows scientists to visualize the data interactively at high quality and make production movies. Finally, this work presents a highly efficient and scalable radiation model based on the sweeping method, which significantly outperforms previous approaches in Uintah, like discrete ordinates. The integrated approach allowed the USC boiler problem to run on 260K CPU cores on Mira.
S. Kumar, J. Edwards, P.-T. Bremer, A. Knoll, C. Christensen, V. Vishwanath, P. Carns, J.A. Schmidt, V. Pascucci.
Efficient I/O and storage of adaptive-resolution data, In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, IEEE Press, pp. 413--423. 2014.
S. Kumar, A. Saha, V. Vishwanath, P. Carns, J.A. Schmidt, G. Scorzelli, H. Kolla, R. Grout, R. Latham, R. Ross, M.E. Papka, J. Chen, V. Pascucci. Characterization and modeling of PIDX parallel I/O for performance optimization, In Proceedings of SC13: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 67. 2013.
Parallel I/O library performance can vary greatly in response to user-tunable parameter values such as aggregator count, file count, and aggregation strategy. Unfortunately, manual selection of these values is time consuming and dependent on characteristics of the target machine, the underlying file system, and the dataset itself. Some characteristics, such as the amount of memory per core, can also impose hard constraints on the range of viable parameter values. In this work we address these problems by using machine learning techniques to model the performance of the PIDX parallel I/O library and select appropriate tunable parameter values. We characterize both the network and I/O phases of PIDX on a Cray XE6 as well as an IBM Blue Gene/P system. We use the results of this study to develop a machine learning model for parameter space exploration and performance prediction.
Keywords: I/O, Network Characterization, Performance Modeling
J.A. Schmidt. Uintah Application Development, SCI Technical Report, No. UUSCI-2008-005, University of Utah, 2008.
C.R. Johnson, D.M. Beazley, Y. Livnat, S.G. Parker, J.A. Schmidt, H.W. Shen, D.M. Weinstein. Applications of Large-Scale Computing and Scientific Visualization in Medicine, SCI Institute Technical Report, No. UUSCI-1997-001, University of Utah, 1997.
C.R. Johnson, D.M. Beazley, Y. Livnat, S.G. Parker, J.A. Schmidt, H.W. Shen, D.M. Weinstein. Applications of Large-Scale Computing and Scientific Visualization in Medicine, In International Journal on Supercomputer Applications and High Performance Computing, 1996.
C.R. Johnson, R.S. MacLeod, J.A. Schmidt. Software Tools for Modeling, Computation, and Visualization in Medicine, In CompMed 94 Proceedings, World Scientific, 1995.
J.A. Schmidt, C.R. Johnson, J.C. Eason, R.S. MacLeod. Applications of Automatic Mesh Generation and Adaptive Methods in Computational Medicine, In Modeling, Mesh Generation, and AdaptiveMethodsforPartial Differential Equations, Edited by I. Babuska and J.E. Flaherty and W.D. Henshaw and J.E. Hopcroft and J.E. Oliger and T. Tezduyar, Springer-Verlag, pp. 367--390. 1995.
J.A. Schmidt, C.R. Johnson. Defibsim: An Interactive Defibrillation Device Design Tool, In IEEE Engineering in Medicine and Biology Society 17th Annual International Conference, IEEE Press, pp. 305--306. 1995.
J.A. Schmidt, C.R. Johnson, R.S. MacLeod. An Interactive Computer Model for Defibrillation Device Design, In International Congress on Electrocardiology, ICE, pp. 160--161. 1995.
D.M. Weinstein, C.R. Johnson, J.A. Schmidt. Effects of Adaptive Refinement on the Inverse EEG Solution, In SPIE, In Experimental and Numerical Methods for Solving Ill-Posed Inverse Problems, Vol. 2570, SPIE, pp. 2--11. 1995.