%----------------------------------------
% -begin computational-experiments.tex
%----------------------------------------
% 
In this section we examine the performance of Uintah's new hybrid CPU-GPU
scheduler and runtime system by running the RMCRT benchmark problem
described by Burns and Christon in \cite{benchmark}.
% which has a constant initial temperature field and the absorption coefficient specified by a analytical function.
This problem is run on a single level using both $41^3$ and $128^3$ cells. In both
cases, the CPU-only version of the \emph{RayTrace()} method consumes more than 90\% of the total compute time.
Significant speedups in this portion of the code yield significant speedups in time to solution.

We choose to use $41^3$ initially so the computed divergence of the heat flux can be compared to the
data published in \cite{benchmark}. For these runs we used 25, 50 and 100 rays per cell.
The testbed RMCRT component was run for 10 timesteps with one patch per core for the CPU implementation and
one patch per GPU for the GPU implementation, with the mean time per timestep computed and compared.
In what follows, we describe the approach taken in the GPU implementation of the ray tracer,
observing the raw speedups obtained, and compare a single Nvidia M2090 GPU against
first a single core and then all cores on a node. These cores were Intel Xeon X5660 (Westmere)
@2.8GHz and AMD Opteron 6200 Series (Interlagos) @2.6GHz for KIDS and TitanDev respectively.
We also examined the scaling behavior of the CPU and GPU implementations.

As mentioned in Section \ref{sec:arches-component}, RMCRT lends itself to scalable parallelism
because the intensities of each ray are mutually exclusive. Therefore, multiple
rays can be traced simultaneously at any given time step in each cell in every Uintah patch.
This leads us to the approach we have taken with the GPU implementation. Our GPU RayTrace kernel uses 
a patch traversal method similar to that used in the existing GPU port of portions of
Uintah's CFD code (ICE algorithm). Here we tile 2D slices of the 3D patch with
2D threadblocks. These slices are in the two fastest moving dimensions (as the patch cells are traversed), X and Y. We assign a single
CUDA thread to each computational cell. Each thread (within a threadblock) is then responsible for tracing the set of rays associated with
its respective cell for each slice. Each thread calculates the sum of the intensities from its set of rays,
and the divergence of the heat flux for the cell, completely independent of
other threads. This avoids potentially costly atomic operations and synchronization.
This approach also allows for a single kernel launch per timestep, avoiding the overhead associated with
multiple kernel launches.

\begin{table}[ht]
\caption{GPU Speedups Relative to CPU Implementation on Single Node of Keeneland and TitanDev}
\begin{center}
\begin{tabular}{|l|c|c|c|c|}
\hline
\multicolumn{5}{|c|}{Single CPU Core vs. Single GPU} \\
\hline
Machine   & Rays   & CPU (s) & GPU (s) & Speedup \\ \hline
Keeneland & 25     & 28.32   & 1.16    & 24.41 \\
\cline{2-5}
1 Core    & 50     & 56.22   & 1.86    & 30.23 \\
\cline{2-5}
Intel     & 100    & 112.73  & 3.16    & 35.67 \\
\hline
TitanDev  & 25     & 57.82   & 1.00    & 57.82 \\
\cline{2-5}
1 core    & 50     & 116.71  & 1.66    & 70.31 \\
\cline{2-5}
AMD       & 100    & 230.63  & 3.00    & 76.88 \\ \hline
\hline
\end{tabular}
% 
\begin{tabular}{|l|c|c|c|c|}
\hline
\multicolumn{5}{|c|}{All CPU Cores vs. Single GPU} \\
\hline
Machine   & Rays   & CPU (s) & GPU (s) & Speedup \\ \hline
Keeneland & 25     & 4.89    & 1.16    & 4.22 \\
\cline{2-5}
12 Cores  & 50     & 9.08    & 1.86    & 4.88 \\
\cline{2-5}
Intel     & 100    & 18.56   & 3.16    & 5.87 \\
\cline{2-5}
\hline
TitanDev  & 25     & 6.67    & 1.00    & 6.67 \\
\cline{2-5}
16 Cores  & 50     & 13.98   & 1.66    & 8.42 \\
\cline{2-5}
AMD       & 100    & 25.63   & 3.00    & 8.54 \\ \hline
\end{tabular}
\label{tbl:single_node_speedups}
\end{center}
\end{table}

Table \ref{tbl:single_node_speedups} shows the relative time to solution for both CPU and
GPU implementations, and the speedups obtained on the single level RMCRT testbed component using
a grid size of $41^3$. These timings were a direct comparison on
a single node of KIDS and TitanDev for 25, 50 and 100 rays per cell. The first set of timings compare
a single CPU core against a single Nvidia M2090 GPU on-node. The second set compare all CPU cores
(12 on KIDS and 16 on TitanDev) with the same single GPU. These results show significant speedups on both machines.

As would be expected, the times to solution
using the GPU implementation for each run are roughly equal for both machines, however
the CPU version of the ray tracer runs considerably faster on Keeneland than TitanDev.
An interesting additional result not shown in Table \ref{tbl:single_node_speedups}, is that
when using all three on-node GPUs on Keeneland and comparing against the CPU implementation, the speedups were
not as significant. We attribute the slowdown to the NUMA and contention effects within the multi-GPU HP SL390
nodes (Figure \ref{fig:hp-sl390}) described in \cite{keeneland_numa}. Currently, our CPU-GPU scheduler
has no notion of GPU affinity. Addressing this issue to maximize utilization of the additional on-node
computational resources in multi-GPU systems will be a focal point in future work.

\begin{figure}
    \includegraphics[width=0.45\textwidth,draft=false]{hp-sl390.jpg}
    \caption{KIDS HP-SL390 Node Configuration}
    \label{fig:hp-sl390}
\end{figure}

Using the CPU-GPU scheduler, we were able to run capability jobs on both machines, using all CPU
cores and GPUs on-node, but saw diminishing returns at larger scale. The all-to-all nature of this problem severely limits the
size of the problem that can be computed, and hence does not yet scale well due to memory constraints with large
highly resolved physical domains.
Figure \ref{fig:scaling} shows strong scaling results for both CPU and GPU implementation on TitanDev.
Similar CPU-only scalability studies of the same single level RMCRT benchmark problem are described in \cite{benchmark}.
It is apparent from the figure that the same scalability breakdown shown in \cite{benchmark} on the XSEDE resource
Kraken, also occurs on TitanDev (the same result was also evident on Keeneland). Figure \ref{fig:scaling} additionally illustrates that the
GPU implementation quickly runs out of work and strong scaling begins breaking down around eight GPUs. Although the mean time
per timestep for the GPU implementation is still considerably lower than the CPU implementation at this point (up to 64 GPUs), ultimately
there is insufficient work, and both implementations suffer from the same exorbitant communication costs
that are the central difficulty in this problem. Addressing this scalability issue will be a primary focus in future work.

\begin{figure}
    \includegraphics[width=0.50\textwidth,draft=false]{scaling_128_100_2.jpg}
    \caption{Strong Scaling Comparison on TitanDev}
    \label{fig:scaling}
\end{figure}

% 
%----------------------------------------
% -end computational-experiments.tex
%----------------------------------------