Exceptions trigger segmentation faults in unit tests (and beyond)

LAMMPS LAMMPS Installation
1

Greetings, I’m trying to build the latest stable LAMMPS on our computer cluster, by adapting the following easybuild configuration to our current toolchain (based on GCC 13.1.0).

LAMMPS-22Jul2025_update4-foss-2025b-kokkos.eb

Output from LAMMPS version:
Large-scale Atomic/Molecular Massively Parallel Simulator - 22 Jul 2025 - Update 4

OS: Linux "Red Hat Enterprise Linux 8.6 (Ootpa)" 4.18.0-372.32.1.el8_6.x86_64 x86_64

Compiler: GNU C++ 13.1.0 with OpenMP 4.5
C++ standard: C++17
Embedded fmt library version: 10.2.0
Embedded JSON class version: 3.12.0

MPI v3.1: Open MPI v4.1.4, package: Open MPI user Distribution, ident: 4.1.4, repo rev: v4.1.4, May 26, 2022

During the testing phase of the build, a number of unit tests are crashing due to an issue with exception handling. This is the command that kicks off the tests:
ctest --no-tests=error -LE unstable -E “TestMliapPyUnified|AtomicPairStyle:meam_spline|KSpaceStyle:scafacos.\*”

The first test that fails is InvalidFlag:

Start   3: InvalidFlag
  3/571 Test   #3: InvalidFlag .....................................***Exception: SegFault  0.76 sec
ERROR: Invalid command-line argument: -xxx (src/lammps.cpp:486)
[hpcl003:3173099:0:3173099] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace (tid:3173099) ====
 0 0x0000000000012ce0 __funlockfile()  :0
 1 0x000000000001d2c8 base_from_object()  /scratch/easybuild/build/GCCcore/13.1.0/system-system/gcc-13.1.0/stage3_obj/x86_64-pc-linux-gnu/libgcc/../../../libgcc/unwind-dw2-fde.c:319
 2 0x000000000001d2c8 linear_search_fdes()  /scratch/easybuild/build/GCCcore/13.1.0/system-system/gcc-13.1.0/stage3_obj/x86_64-pc-linux-gnu/libgcc/../../../libgcc/unwind-dw2-fde.c:969
 3 0x000000000001df41 find_fde_tail()  /scratch/easybuild/build/GCCcore/13.1.0/system-system/gcc-13.1.0/stage3_obj/x86_64-pc-linux-gnu/libgcc/../../../libgcc/unwind-dw2-fde-dip.c:519
 4 0x000000000001df41 _Unwind_Find_FDE()  /scratch/easybuild/build/GCCcore/13.1.0/system-system/gcc-13.1.0/stage3_obj/x86_64-pc-linux-gnu/libgcc/../../../libgcc/unwind-dw2-fde-dip.c:573
 5 0x00000000000195ca uw_frame_state_for()  /scratch/easybuild/build/GCCcore/13.1.0/system-system/gcc-13.1.0/stage3_obj/x86_64-pc-linux-gnu/libgcc/../../../libgcc/unwind-dw2.c:1005
 6 0x000000000001b03d _Unwind_RaiseException()  /scratch/easybuild/build/GCCcore/13.1.0/system-system/gcc-13.1.0/stage3_obj/x86_64-pc-linux-gnu/libgcc/../../../libgcc/unwind.inc:104
 7 0x00000000000bc62b __cxa_throw()  /scratch/easybuild/build/GCCcore/13.1.0/system-system/gcc-13.1.0/stage3_obj/x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/../../../../libstdc++-v3/libsupc++/eh_throw.cc:93
 8 0x0000000000c450d1 LAMMPS_NS::Error::universe_all()  error.cpp:0
=================================
[hpcl003:3173099] *** Process received signal ***
[hpcl003:3173099] Signal: Segmentation fault (11)
[hpcl003:3173099] Signal code:  (-6)
[hpcl003:3173099] Failing at address: 0x699700306aeb
[hpcl003:3173099] [ 0] /lib64/libpthread.so.0(+0x12ce0)[0x15554e60ace0]
[hpcl003:3173099] [ 1] /cm/shared/uniol/sw/zen4/13.1/GCCcore/13.1.0/lib64/libgcc_s.so.1(+0x1d2c8)[0x15554e21a2c8]
[hpcl003:3173099] [ 2] /cm/shared/uniol/sw/zen4/13.1/GCCcore/13.1.0/lib64/libgcc_s.so.1(_Unwind_Find_FDE+0xd1)[0x15554e21af41]
[hpcl003:3173099] [ 3] /cm/shared/uniol/sw/zen4/13.1/GCCcore/13.1.0/lib64/libgcc_s.so.1(+0x195ca)[0x15554e2165ca]
[hpcl003:3173099] [ 4] /cm/shared/uniol/sw/zen4/13.1/GCCcore/13.1.0/lib64/libgcc_s.so.1(_Unwind_RaiseException+0x1cd)[0x15554e21803d]
[hpcl003:3173099] [ 5] /cm/shared/uniol/sw/zen4/13.1/GCCcore/13.1.0/lib64/libstdc++.so.6(__cxa_throw+0x3b)[0x15554f6bf62b]
[hpcl003:3173099] [ 6] /scratch/easybuild/build/LAMMPS/22Jul2025_update4/foss-2023a-kokkos-test/easybuild_obj/liblammps.so.0(+0xc450d1)[0x1555506b00d1]
[hpcl003:3173099] *** End of error message ***

Besides #3, the following tests report SegFault:

  • #4: Tokenizer
  • #7: ArgUtils
  • #8: Utils
  • #10: Lepton
  • #11: FmtLib
  • #14: AtomStyles
  • #16: InputConvert
  • #17: MoleculeFile
  • #19: PotentialFileReader
  • #21: TextFileReader
  • #22: FileOperations
  • #23: DumpAtom
  • #24: DumpAtomGZ
  • #25: DumpCustomGZ
  • #26: DumpCfgGZ
  • #27: DumpLocalGZ
  • #28: DumpXYZGZ
  • #35: DumpCfg
  • #38: SimpleCommands
  • #39: LatticeRegion
  • #40: Groups
  • #41: Regions
  • #42: DeleteAtoms
  • #43: SetProperty
  • #44: Labelmap
  • #45: Variables
  • #46: KimCommands
  • #47: ResetAtoms
  • #48: ComputeGlobal
  • #54: LibraryProperties
  • #55: LibraryObjects
  • #65: AdvancedUtils
  • #66: ErrorClass
  • #72: FortranProperties
  • #84: PythonOpen
  • #85: PythonCommands
  • #86: PythonNumpy
  • #167: MolPairStyle:lepton

Despite these Segmentation Faults in the testing, basic functionality, like running the lowercase examples, seems to work.

At the moment, I do not understand what is causing the SegFaults in the unit tests, so I am not sure what to change to fix the build. Any advice would be appreciated.

2

@gcoleman Here are some suggestions:

  • to debug the cause of the test failures, drop easybuild and compile LAMMPS manually
  • do not enable any packages and turn off BUILD_MPI, BUILD_OMP, and BUILD_SHARED_LIBS only enable BUILD_TESTING
  • use -D CMAKE_BUILD_TYPE=Debug or -D CMAKE_BUILD_TYPE=RelWithDebInfo (the default)
  • try with the ‘maintenance’ branch instead of ‘stable_22Jul2025_update4’

If this still fails, try with a different compiler (clang or different GCC version).
As a last resort, there is a validated apptainer/singularity container available at: https://download.lammps.org/static/fedora41_musl_mingw.sif

If it passes, re-enable the disabled features one by one enabling the basic package preset as penultimate and the most preset as ultimate option and check when the segfault returns.

Keep us posted about your findings.

Thanks,
Axel.

P.S.: I am very close to publishing 22Jun2025_update5 with many backported bug fixes, so I am particularly keen on knowing about problems with the maintenance branch.

1 Like
3

I forgot one thing,

For some of the simple tests, you can also run them without ctest and any environment or command line flags under the debugger:

gdb --args ./test_tokenizer
[...]
(gdb) run
[...]
[ RUN      ] ArgInfo.bad_idx2
[       OK ] ArgInfo.bad_idx2 (0 ms)
[----------] 34 tests from ArgInfo (0 ms total)

[----------] Global test environment tear-down
[==========] 34 tests from 1 test suite ran. (0 ms total)
[  PASSED  ] 34 tests.
[Inferior 1 (process 3158317) exited normally]

this works well for many of the simple tests listed at the beginning like test_tokenizer, test_arguilts, test_utils, test_file_operations and so on.

You can also intercept exceptions with catch throw at the gdb prompt, so you can inspect the cause of failure from the stack.

4

@gcoleman Do you have any updates?
Did you find the cause for the segfaults?

If there is any issue on the LAMMPS side, I would like to know about it rather sooner than later, so we can have it fixed for the upcoming stable release for 2026.

Thanks,
Axel.

5

Dear @akohlmey, thank you for the debugging suggestions.

I actually spent some time differentially analyzing the enabled packages and was able to remove a package that caused some errors, but not the one triggered by invalid argument. Then, afterwards, I tried to repeat this on another installation, and kept running into the same problem. Confusingly, this was fixed by disabling the shared library in the build: -DBUILD_SHARED_LIBS=off' and then reënabling it–the easybuild install script was relying on the presence of a shared library to copy to the python extension directory. So, I currently have a working installation of LAMMPS-22Jul2025_update4.

Following your suggestion, I tried doing basic CMake builds for maintenance: dfa0051 and stable: 611ca3b, with mostly default options but with shared libraries enabled. Neither of these reproduced the problem that I was having with those builds.

Perhaps something within the more complicated easybuild environment, or something specifically in our cluster toolchain, was responsible for the problem with those builds. But I haven’t managed to reproduce the error yet. Presumably I could do so by continuing to augment the build environment and/or CMake configuration until it resembles what is in the easybuild configuration.

Sorry I didn’t come up with something more concrete. Best regards, G