Finite Element Large Eddy Simulation


Principal Investigator

Hugh Blackburn

Building, Construction and Engineering,

CSIRO Division

The project is concerned with developing techniques for large eddy simulation of bluff-body flows and
flows in complex geometries. These flows have a wide range of application in mechanical, civil and offshore engineering, and at present the only available approaches for engineering investigations are Reynolds-averaged simulations or physical experiments. The Reynolds-averaged approach is very often inaccurate even in modelling the time-mean flow, and fluctuating quantities are very poorly predicted. Experimental methods are becoming increasingly expensive, while computer simulations, using large eddy simulation, are just becoming feasible for these kinds of problems. Finite element based methods required for complex geometries have not yet been extensively employed for large eddy simulation, due to the algorithmic difficulties encountered on typical supercomputer architectures, and the complexity of sub-grid scale stress models suitable for arbitrary geometries.



Dean Niclasen

Mechanical Engineering,

Monash University



g89 - VPP



What are the results to date and the future of this work?

Following the initial application both investigators attended the ANUSF course on vectorization and tuning for the VPP. As a result of information and assistance received there and subsequently, porting and vectorization of existing low-order finite element and high order (spectral) finite element unsteady Navier­Stokes codes was carried out. At that time the low-order code had 30% vectorization on test problems, while the spectral element code achieved 50% for typical 3D jobs. Further vectorization of the spectral element code was somewhat impeded by absence of a native C++ compiler ­ runtime information provided by system sampling utilities reports time spent in top-level routines only, and debugging tools fail to work ­ C++ is used only for the top level of the code: compute-intensive parts are carried by FORTRAN & C routines.

It was then planned to use MPI as a basis for parallelization of the spectral element code, since the spatial discretization employed (2DxFourier) is well suited to the coarse-grained architecture of the VPP. This work has now been successfully carried out, together with further vectorization of gradient operator kernels to improve performance in computation of nonlinear terms and PCG solution methods. Coding of eddy-viscosity type LES has also been incorporated and is undergoing testing.

Appendix B -


The algorithm employed has proven to be quite efficient in concurrent execution, as anticipated: solutions for each Fourier mode can (if possible) execute on a separate processor, with interprocessor communication required only during computation of nonlinear terms in the Navier­Stokes equations.

Sample speedup comparisons for a medium-sized (120000-node/100MB) 3D problem, based on elapsed times:

 #procs time speedup efficiency CPUtime %Vector
1 972 1 1 880 46
2 540 1.8 .90 873 40
4 340 2.9 .71 1098 32

Overall performance for these runs was approximately 2.5 times that of a 4-processor 250Mhz DEC Alphaserver. Since the comparison were carried out, per-node performance has been significantly enhanced by vectorization of 2D field-gradient operators which then consumed of the order 50% of execution time: typical performance figures for the kernel routines are now between 500 & 1000 Mflops. Concurrent efficiency is likely to improve as message sizes increase for larger problems, possibly also with a new memory-transpose algorithm if required.

In terms of algorithm development, it is planned to develop a new inter-processor memory transpose based on recursive doubling in order to remove potential bottlenecks, if this proves to be a significant problem as the number of PEs employed increases.

Futher grants of time will be consumed mainly in parallel production runs, initially to investigate DNS & LES of fluid-structure interaction problems.

What computational techniques are used?

The method is a spectral element/Fourier incompressible flow solver, which employs a time-splitting algorithm based on backwards differentiation. Two-dimensional planes of data are decomposed into high-order ("spectral") elements which employ orthogonal functions, such as Legendre or Chebyshev polynomials, as basis functions. Extension to three spatial dimensions is achieved using Fourier expansions, which also allows most of the operations in each timestep to be carried out concurrently. Nonlinear terms in the Navier-Stokes equations are are computed pseudo-spectrally in physical space, and each of the products requires an inter-processor data transpose of velocity field components. The remaining two-dimensional Poisson/Helmholtz problems which comprise the time-split are carried out concurrently. Owing to the use of orthogonal basis functions in all directions, spatial converegence of the method depends exponentially on the number of grid points used.

- Appendix B