The analysis of complex physical and biological systems necessitates the accurate resolution of in-teractions across multiple spatio-temporal scales, the consistent propagation of information between concurrently coupled multi-physics processes, and the effective quantification of model error and para- metric uncertainty. Addressing these grand challenges is a multi-faceted problem that poses the need for a highly sophisticated arsenal of tools in stochastic modeling, high-performance parallel computing, and probabilistic machine learning. Through the lens of three realistic large-scale applications, this talk aims to demonstrate how the compositional synthesis of such tools is introducing a new paradigm in scientific discovery. First, we present multi-scale blood flow simulations in the human brain, and show how high-order methods, massively parallel computing, and concurrent coupling of multi-physics solvers can uncover intrinsic physiological mechanisms in health and disease. We will demonstrate how the introduction of probabilistic machine learning techniques and the key concept of multi-fidelity modeling provide a scalable platform for information fusion and lead to significant computational affordability and expediency gains. The second application involves an environmental study that illustrates how machine learning tools enable the synergistic combination of simulations, noisy measurements and em- pirical models towards quantifying the anthropogenic effect in the increasing acidification of coastal waters, and developing a cost-effective monitoring and prediction mechanism. Lastly, we consider the shape optimization of super-cavitating hydrofoils of an ultrafast marine vessel for special naval oper- ations. Specifically, we show how the combination of turbulent multi-phase flow simulations and the concept of multi-fidelity Bayesian optimization allows us to tackle complex engineering design problems in which a rigorous assessment of uncertainty and risk becomes critical in policy and decision making.