%----------------------------------------
% -begin introduction.tex
%----------------------------------------
% 
An important trend in high performance computing is the planning and design 
of software framework architectures for emerging and future systems with multi-petaflop and
eventually exaflop performance. With ever imposed demands on system architects
for increased density and power efficiency, traditional systems are now being
augmented with an increasing number of graphics processing units
(GPUs)\cite{keeneland_numa}. This design is most notable in systems such as the
Keeneland Initial Delivery System (KIDS)\footnote {KIDS is an experimental
HP-Nvidia GPU cluster located at the National Institute for Computational Sciences
with 120 compute nodes, each with two Intel Xeon X5660 (Westmere 6-core @2.8GHz)
processors, 24GB memory, InfiniBand QDR (single rail) interconnect and 3 Nvidia
Tesla M2090 GPUs.} \cite{keeneland_webpage}. This architectural trend
is also evidenced in the current upgrade path of the DOE
Jaguar\footnote{Jaguar is a DOE supercomputer located at the Oak Ridge National
Laboratory with 18,688 compute nodes each of which contains a single 16-core AMD
Opteron 6200 Series (Interlagos cores @2.6GHz) processor on one of its two
sockets, 32GB memory and Gemini interconnect, giving 299,008 processing
cores. Currently on 960 nodes, the second socket contains a single Nvidia
Tesla 20-series GPU. This 960 node partition is known as TitanDev.} system to Titan \cite{titandev_webpage}.

Significant challenges face those trying to program for such architectures. The
first of these challenges is the prospect of significantly less memory per core
as the numbers of cores per socket continues to grow. In order to address this challenge,
as recognized by a number of authors \cite{Ott2008,Balaji2008},
Uintah \cite{TGRID10MB}, an open-source software framework has moved from a model that only
uses MPI to one that employs MPI to communicate between nodes and a shared memory
model using Pthreads to map the work onto available cores in a node\cite{Qingyu-TG11}. The Uintah
task-based model lends itself better to the use of Pthreads
rather than OpenMP. This approach has led to the development of a multi-threaded MPI
runtime system, including a threaded task scheduler that has enabled Uintah to
show excellent strong and weak scaling up to 196K cores on the DOE Jaguar XT5
system and good initial scaling to 262k cores on the upgraded DOE Jaguar
XK6 system \cite{John-UUSCI-2012-002}. Using this approach has reduced Uintah's
on-node memory usage by up to 80\%\cite{Qingyu-TG11}.

A second challenge posed by such architectures is the design of runtime systems
that maximize system utilization by fully exploiting all available processing
resources on-node. Central to this goal is overcoming the inherent bandwidth
bottleneck of PCI-express (PCIe) transfers to-and-from the GPU, as discrete GPUs
are typically hosted in PCIe slots. Data copies across the PCIe bus, which has
a maximum theoretical bandwidth of 8.0GB/s (PCI Express Gen2 x16 for the Nvidia
Tesla C20 series cards). In practice, this rate is closer to 3.3GB/s
when using paged memory, and 5.3GB/s using pinned (page-locked) memory. For
memory bandwidth bound tasks, this bottleneck requires more advanced techniques
to harness the computational power offered by GPUs. Many current approaches to
this problem leave CPU cores idle during GPU-based computation, and others simply
do not extend focus beyond a single GPU. These approaches waste substantial
available computational power.

Uintah is novel in its use of a asynchronous, task-based paradigm, with complete
isolation of the application developer from parallelism. The individual tasks are viewed as part of
a directed acyclic graph (DAG) and are executed adaptively, asynchronously and
often out of order \cite{Qingyu}. Uintah uses a novel adaptive meshing
approach \cite{IPDPS10}  as well as a variety of fixed mesh and particle solution methods.

In this paper we look at how to extend Uintah's hybrid multi-threaded MPI runtime
system\cite{Qingyu-TG11} to support, schedule and execute both GPU and CPU tasks simultaneously.
We examine the design of a CPU-GPU scheduler in the context of a developing
scalable hierarchical ray-tracing radiation transport model to provide Uintah with
additional capabilities for heat transfer, and electromagnetic wave propagation.
This work directly addresses the second major challenge introduced by heterogeneous
systems, specifically utilizing all processing resources available on-node.
In what follows  Section 2 provides an overview of the Uintah software, while Section
3 describes ARCHES, the Uintah component designed for simulation of turbulent reacting
flows with participating media radiation, and its ray-tracing radiation transport model
for which we are developing GPU-based capabilities.
Section 4 briefly describes Uintah's multi-threaded MPI runtime system design\cite{Qingyu-TG11},
which this work extends. The new CPU-GPU task scheduler design and the multitude of techniques
used to overlap PCIe transfers and MPI communication with GPU and CPU computation are also
described in Section 4. Finally, in Section 5, we describe computational experiments
that illustrate the effectiveness of our new hybrid CPU-GPU task scheduler over a range
in scales of processor core numbers and GPUs, comparing results with and without GPUs on KIDS and
TitanDev, the 960 node partition of the Jaguar system outfitted with GPUs.
The paper concludes by describing future work in this area.
% 
%----------------------------------------
% -end introduction.tex
%----------------------------------------