Hybrid Programming in CUDA, OPENMP and MPI.
Under Construction!
CUDA (an acronym for Compute Unified Device Architecture) is a parallel computing architecture developed by NVIDIA. CUDA is the computing engine in NVIDIA graphics processing units or GPUs that is accessible to software developers through industry standard programming languages. A compiled CUDA program can therefore execute on any number of processor cores, and only the runtime system needs to know the physical processor count.
In contrast, Open Multi-Processing (OpenMP), a kind of shared memory architecture API, provides a multithreaded capacity. The OpenMP (Open Multi-Processing) is an application programming interface (API) that supports multi-platform shared memory multi-processing programming in C, C++ and FORTRAN on much architecture, including UNIX and Microsoft Windows platforms. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior.
Message Passing Interface (MPI) is a specification for message passing operations. It defines each worker as a process. MPI is currently the de-facto standard for developing HPC applications on distributed memory architecture. It provides language bindings for C, C++, and FORTRAN. MPI offers portability, standardization, performance, and functionality, and includes point-to-point message passing and collective (global) operations, all scoped to user-specified groups of processes.