I am implementing finite element code using MPI. Obviously, when using MPI I will have to include communication between different nodes so that all processors have all relevant information. As this is my first time doing something like this, I have implemented a very naive algorithm to get updated volume information for each element as shown below:
!loop over all of the processors do n=0,numProcs-1 !for each iteration of the loop, one processor will hand out relevant information if( my_id .eq. n ) then do i=0,numProcs-1 if( i .ne. my_id ) then currProc=i !get the number of volumes processor i needs from processor n call MPI_RECV( numNeeded,1,MPI_INTEGER,i,my_id, & MPI_COMM_WORLD,status,ierr ) !for each volume that processor i needs: do j=1,numNeeded !get the element number call MPI_RECV( currNeeded,1,MPI_INTEGER,i,j, & MPI_COMM_WORLD,status,ierr ) !return the volume of that element call MPI_SEND( myElements(elementGlobalToLocal(currNeeded))%xsj,1, & MPI_REAL8,i,j,MPI_COMM_WORLD,status,ierr ) enddo endif enddo else !tell processor n how many volumes are needed from it, calculated elsewhere call MPI_SEND( numElementsNeeded(n+1),1,MPI_INTEGER,n,n, & MPI_COMM_WORLD,status,ierr ) !for each volume needed do j=1,numElementsNeeded(n+1) !send the element number, calculated elsewhere call MPI_SEND( elementsNeeded(n+1, j),1,MPI_INTEGER,n,j, & MPI_COMM_WORLD,status,ierr ) !receive the appropriate volume call MPI_RECV( myExternalXsj(currReceived),1,MPI_REAL8,n,j, & MPI_COMM_WORLD,status,ierr ) !global element numbering is 1-numTotElems !local numbering is 1-numElemsPerProc !elementGlobalToLocal keeps track of conversion from local to global elementGlobalToLocal(elementsNeeded(n+1, j))=currReceived currReceived=currReceived+1 enddo endif enddo
While this code does give the right results (i.e., all of the processors have all the information they need, and the values passed are the correct values), the performance is abysmal. Even when using
-np 2, roughly 50% of the total time spent on volume calculations is spent doing this communication; when using
-np 8 that comes up to a startling 94%. The major issue I see with the code above is that it doesn't scale very well; when process 0 is communicating to process 1, processes 2 and 3 are just hanging, which will cause a significant slowdown.
What would be a good way to address this? Also, are there any other major issues with the code that I am not seeing?