[Getdp] Parallelization patch for GetDP with PETSc

[Davide Rondini]

Hello,

I developed a patch for GetDP to enable it to work properly with parallel 
version of PETSc. Essentially, I removed a couple of deadlocks, and introduced 
the check for MatGetOwnershipRange in functions which set the elements of 
matrices and vectors. This ensures that everything is assembled the right way 
even using mpiaij matrices.

I tested it using some sample tests taken from wiki and I've seen that it 
works fine. Accuracy is ok, it is within tolerance set between serial and 
parallel runs (also compared with sparsekit version). Performance scale though 
number of processors as expected.

Anyway, this is a crude patch, it has 2 disadvantages:

1. The check of competence of an element of the matrix to each processor is 
done inside LinAlg functions, which forces all processors to check all 
elements of the matrix. This is not efficient, this operation should be made in 
the cycle that assemples the matrix, but this is not so simple, it requires to 
modify GetDP code in multiple points. I hope to have time to remove this 
overhead in the future. It wolud be very helpful if someone can show me where 
the code has to be modified.

2. The parallel process works fine up to the calculation phase, while post-
processing has some problems to run in parallel. This is because PETSc do not 
allow to access the elements of a vector or matrix which pertain to another 
processor. So, performing post operations in serial mode inserts spurious 
zeros or nan in the solution, and doing parallel post processing generates 
messy output file. Implementing parallel I/O is long and complex, for me it is 
sufficient to run resolution and post-operation separately, this way:

mpirun -n X getdp simulation.pro -solve Resolution
mpirun -n 1 getdp simulation.pro -pos PostOperation

this workaround works fine.

I hope this can be helpful.

Regards,
Davide Rondini
-------------- next part --------------
A non-text attachment was scrubbed...
Name: getdp_1.2.1_parallel.patch
Type: text/x-patch
Size: 20119 bytes
Desc: not available
URL: <http://www.geuz.org/pipermail/getdp/attachments/20090603/94295e2d/attachment.patch>