I am trying to put together a preliminary benchmark suite of CUDA applications to address the current inability of researchers to evaluate architecture and compiler optimizations for GPUs; the current state of the practice is to run the SDK examples and any hand coded applications that one has lying around. It is hard to determine how effective a given optimization is or to verify the results reported by a research group because no one has access to the same set of applications, and it is not clear if anyone actually cares about the applications being tested.
The ideal workload should be representative of the most important applications for general purpose computing using CUDA in the same way that SPEC/PARSEC/SPLASH/MediaBench provide a basis for comparing optimizations for CPUs. It should draw the most representative sections from widely used applications and algorithms written in CUDA. All of the applications should be open source with a free for commercial use license.
Here is a preliminary list of application domains that I propose are important drivers for a benchmark suite for CUDA applications:
1) High Performance Computing A) Linear Algebra i) QR decomposition (?) ii) SV decomposition (?) B) Solvers i) Some PDE solver (?) ii) Graph Partitioning (?) iii) K-Nearest Neighbor (?) C) Signal Processing i) FFT/DCT (CUFFT/VSIPL) ii) Convolution/FIR/IIR Filters (VSIPL) 2) Media/Graphics/Image Processing A) Video Encoding/Decode i) H264 (?) B) Graphics i) Ray Tracing (?) ii) others? 3) Financial / Data Processing A) Option Pricing / Financial Simulation i) Black Scholes (many examples) ii) Collateralized Debt Obligation (?) iii) Portfolio Risk Analysis (?) B) Compression/Encryption i) File compression (DEDUP?) ii) AES? C) Database operations i) Joins (Relation Joins on GPUs) ii) Reductions/Prefix Sum (Many Examples) iii) Sorting (GPUQuicksort) 4) Machine Learning / AI 5) Simulation A) Physics i) Particles/Fluids/Aerodynamics (PhysX maybe) B) Digital/Analog Hardware i) VHDL/Verilog/Netlist simulation (?) ii) E&M Wave Propagation (?) iii) SPICE/etc (?) C) Chemistry/Biology i) Molecular Dynamics (NAMD) D) Astrophysics i) Nbody Simulation (?)
If anyone has any suggestions as to domains that I have missed or specific CUDA apps that fit into a particular domain, please let me know. If you are a researcher developing optimizations for CUDA applications, what would make it easier to use such a benchmark suite for any results that you publish? If you are an application developer, would you be willing to contribute either full CUDA applications or representative benchmarks in the hope that future improvements to GPUs or compilers will accelerate your application?
If you know of any open source CUDA applications that fit into these domains, would you please reply with a link?