# MPI Examples: Ghost Exchange

Welcome to the Ghost Exchange example directory! 

This example is intended to be a simplified instance of what an actual scientific application code using the message passing interface (MPI) might look like. We consider a problem that is discretized on a structured Cartesian grid where the solution field is defined on a cell-wise fashion, meaning that there is a 1-1 correspondence between the computational cells used to discretize the domain and the discrete values of the solution field: the solution is defined on the cells. MPI is used to distribute the necessary computational workload in parallel, with multiple processes handling partitions of the initial computational domain and carrying out the necessary work in parallel on their assigned partition: each process is assigned a subset of the computational domain. This partition is a subset of the total number of cells used to discretize the domain. As the reader can guess, the problem has to be designed and implemented in a way that such parallel partitioning is indeed possible and correct in terms of the final answer, which means that the solution obtained in serial needs to be the same (up to machine precision) as the one obtained in parallel using MPI. From a mathematical standpoint, the parallel computation relies on the so called <i>ghost cells</i> which are cells assigned to a given process that are instead own by a different one: this means that a process can read the solution defined on the ghost cells but should not modify it. Hence, the partitioning of the computational domain among processes is actually overlapping, due to the presence of the ghost cells. It would  not be overlapping considering only the cells owned by a process. The ghost cells exist because mathematical operators in discretized form need information on the neighboring cells to compute the values of physical fields on a given cell. Ghost cells surround the cells owned by a process, therefore forming a <i>halo</i> around the subdomain owned by the process. Depending on the depth of the mathematical operator, the halo might be one or several layers thick. In this example, the mathematical operator is a five point average, hence only one halo layer is necessary.  To correctly compute the value of the solution on the cells owned by a given process, the information on the ghost cells needs to be used, but it also needs to be used with the correct values. These correct values need to be transferred from the process that owns the ghost cells, to the process that is only using them to update the solution on the cells it owns. This is done through a so called <i>halo exchange</i>, also called halo update: this is an operation done using MPI where, the solution values on the halo cells are exchanged between processes to make sure that they are current by the time they are used to update the values of the solution on the owned cells.

We call this example Ghost Exchange because MPI is used to perform the exchange of information on the ghost cells through the implementation of halo exchanges. The actual computation of the solution done by a given process may be done in parallel with the use of OpenMP or HIP, for instance.

To summarize, let's go over the main steps in the Ghost Exchange example code:
1. Initialization: in this step, the initial values of the solution field on the cells are prescribed. Then the boundary conditions are enforced, and a halo exchange is performed, so that the solution at the halo of each process is up to date.
2. Time stepping: the time stepping procedure just consists of successively calling a five point averaging operator for as many times as the number of iterations prescribed (default is 1000). Within each time step, we call the averaging operator, then once again enforce the boundary conditions and perform a halo exchange to update the solution at the halo for the next step.
3. Output: the final step is to output the timings of the solution advancement, boundary condition enforcement and ghost exchange, to measure how long each of these steps took compared to the others.

There are three major versions of a Ghost Exchange example:
- [`GhostExchange_ArrayAssign`](https://github.com/amd/HPCTrainingExamples/tree/main/MPI-examples/GhostExchange/GhostExchange_ArrayAssign): this version considers a 2D domain where parallelism is further enhanced with the use of OpenMP. OpenMP is used to perform the necessary computations in parallel. MPI is used to perform the data exchange.
- [`GhostExchange_ArrayAssign_HIP`](https://github.com/amd/HPCTrainingExamples/tree/main/MPI-examples/GhostExchange/GhostExchange_ArrayAssign_HIP): this version also considers a 2D domain but HIP is used in place of OpenMP to further enhance parallelism. MPI is used for data exchange.
- [`GhostExchange3D_ArrayAssign`](https://github.com/amd/HPCTrainingExamples/tree/main/MPI-examples/GhostExchange/GhostExchange3D_ArrayAssign): this version is similar to `GhostExchange_ArrayAssign` but instead considers a 3D domain. Currently, only a basic implementation is available and the performance improvements done for `GhostExchange_ArrayAssign` are not implemented at the moment.

The reader is encouraged to explore the single directories for further details on the code and various subversions.
