[Getdp] large problems with getdp
Guillaume Demesy
gdemesy at physics.utoronto.ca
Tue Dec 14 15:56:42 CET 2010
Hello again Helmut,
I had issues similar to yours when I started with getdp... UMFPACK
often crashed on my machines at some load thresold. I don't really have
an explanation, sorry.
Then, I compiled getdp against petsc with MUMPS. It works really well,
even with problems as big as millions of DOF requiring tens of gigs of
memory (then it's better if you build the parallel version, though), as
reported by the developers.
The time spent on the compilation of petsc/getdp really worth it, IMO...
Here is how I compiled my last getdp:
export PETSC_DIR=yourPETCdir PETSC_ARCH=yourPETSCarch
./configure \
--with-cc=/yourMPIpath/bin/mpicc \
--with-fc=/yourMPIpath/bin/mpif90 \
--with-f77=/yourMPIpath/mpif77 \
--with-cxx=/yourMPIpath/mpicxx \
--with-petsc-prefix=$PETSC_DIR \
--with-petsc-arch=$PETSC_ARCH \
--with-gsl-prefix=$GSL_BASE
--enable-petsc \
--enable-slepc \
--disable-sparskit \
--disable-arpack \
Hope this helps,
Guillaume
On 14/12/2010 09:31, Helmut Müller wrote:
> Hello Guillaume,
>
> thank you for your reply.
>
> Yes, problems with 200000 DOF can be solved on this machine, without
> probs. But larger problems with > 250.000 DOF always lead to malloc
> failure messages. I assume not the machine is limiting the size of
> solvable probs, but getdp, at least in the precompiled dl version.
> Unfortunately I didn´t manage to recompile getdp by myself. I have had
> different tries in the past, but no success, so I use the downloadable
> version.
>
> Perhaps I´ll have to dig myself into recompilation again. Can you
> confirm that a recompiled version would solve larger problems than the
> precompiled version? Next question :-) why ? And then, why ist the
> precompiled Version limited ?
>
> Please, do not missunderstand me, I don´t want to complain about this
> software or the service that precompiled Versions are availlable, but
> I´d like to understand ( typical engeneer ).
>
> Thanks in advance, best regards,
>
> Helmut Müller
>
> PS: just tried your suggested command with getdp version 2.1.0 :
>
> Info : InitSolution[T]
> Info : Generate[T]
> Info : (CPU = 13.1382s, Mem = 537Mb)
> Info : Solve[T]
> Info : N: 225215
> getdp(19601) malloc: *** mmap(size=2147483648) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
>
> ( this message repeated several times)
>
> getdp(19601,0xa052c500) malloc: *** mmap(size=1948217344) failed
> (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
>
> UMFPACK V5.2.0 (Nov 1, 2007): ERROR: out of memory
>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Error in external library!
> [0]PETSC ERROR: umfpack_UMF_numeric failed!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 7, Mon Jul 6
> 11:33:34 CDT 2009
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: problem.pro on a umfpack-c named MacBookPro.local by
> helmutmuller Tue Dec 14 15:26:14 2010
> [0]PETSC ERROR: Libraries linked from
> /Users/geuzaine/src/petsc-3.0.0-p7/umfpack-cxx-opt/lib
> [0]PETSC ERROR: Configure run at Tue Aug 18 10:37:03 2009
> [0]PETSC ERROR: Configure options --with-debugging=0
> --with-scalar-type=complex --with-clanguage=cxx --with-shared=0
> --with-x=0 --with-fortran=0 --with-mpi=0 --with-umfpack=1
> --download-umfpack=ifneeded
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: MatLUFactorNumeric_UMFPACK() line 176 in
> src/mat/impls/aij/seq/umfpack/umfpack.c
> [0]PETSC ERROR: MatLUFactorNumeric() line 2390 in
> src/mat/interface/matrix.c
> [0]PETSC ERROR: PCSetUp_LU() line 222 in src/ksp/pc/impls/factor/lu/lu.c
> [0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: User provided function() line 62 in
> unknowndirectory/LinAlg_PETSC.cpp
> Abort trap
>
> So I was unable to solve a problem wit 225215 DOF. So, how the hell
> have you ( or Christophe) been able to solve about 10 millions DOF ?
>
> regards, Helmut
>
>
>
> Am 14.12.2010 um 14:31 schrieb Guillaume Demesy:
>
>> Hello Helmut,
>>
>> 8Gb should be more than enough to solve this 200000 DOF problem.
>> Getdp binaries are compiled with the PETSc solver umfpack. Have you
>> tried it? Or are you using GMRES?
>> getdp myfile.pro -solve myresolution -ksp_type preonly -pc_type lu
>> -pc_factor_mat_solver_package umfpack
>>
>> But the best solution when tackling big problems is probably to
>> 1- recompile your own petsc with openmpi support and MUMPS
>> 2- compile getdp with your petsc
>> ...which will provide you a parallel 'solve'. The preprocessing will
>> remain serial.
>>
>> see:
>> https://geuz.org/trac/getdp/wiki/PETScCompile
>> ./configure --CC=/opt/local/bin/openmpicc
>> --CXX=/opt/local/bin/openmpicxx --FC=/opt/local/bin/openmpif90
>> --with-debugging=0 --with-clanguage=cxx --with-shared=0 --with-x=0
>> --download-mumps=1--download-parmetis=1--download-scalapack=1
>> --download-blacs=1 --with-scalar-type=complex
>>
>> Good luck!
>>
>> Guillaume
>>
>>
>>
>> On 14/12/2010 06:28, Helmut Müller wrote:
>>> Hi all,
>>> first I´d like to thank you for this impressive software!
>>> I use it for (quite simple) simulations regarding buildingphysics, I just solve heat equations. Therefore I have quite large models (2m by 2m by 2m) with some small parts or thin layers (10mm).
>>> Unfortunately I have to spend a lot of time adjusting the mesh and/or simplify the geometry because I didn´t manage to solve Problems with more than approx. 220.000 DOF on my Mac (8GB RAM, quadCore ). Those problems are solved within seconds or few minutes.
>>> From working with other FEM Simulations I know that it is really important to have a "good" mesh, but I´d like to spend less time for optimization of the geometry and/or the mesh for the price of longer calc times on larger models. A longer calculation time
>>> would cost me far less than optimization.
>>> In this context I have read a mail on the list:
>>> > This has been successfully tested with both iterative (GMRES+SOR) and
>>> > direct (MUMPS) parallel solvers on up to 128 CPUs, for test-cases up to
>>> > 10 millions dofs.
>>> With which trick or procedure has this been done ? On which Platform ? How can I at least use the available memory to perform such calculations ( my getdp 2.1 on MacOS (binary download) uses only a small part ca. 1GB of the available memory, pets fails with
>>> a malloc error message, the new release of getdp uses all cores but with no benefit for maximum possible model size in respect to DOF. So I assume with 8GB it should be possible to do calculations of at least 500000 DOF.
>>> So, what do I miss ? Could partitioning of the mesh and doing separate, iterative calculations be a solution ?
>>> Thanks in advance for any suggestion. I assume that other people are interested in this too.
>>> Helmut Müller
>>>
>>>
>>> _______________________________________________
>>> getdp mailing list
>>> getdp at geuz.org
>>> http://www.geuz.org/mailman/listinfo/getdp
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.geuz.org/pipermail/getdp/attachments/20101214/b42feddc/attachment.html>