Capture Linux core dumps and validate the debugging chain

When a production process exits unexpectedly, restarting the service restores operation but does not explain the crash. If the logs contain only a signal number, diagnosis still requires the call stack at the time of the crash and the matching build. This walkthrough sets up controlled core dump collection on Ubuntu Server 24.04, validates it with an isolated process, and only then considers enabling it for production services.

AMDSSH dedicated servers are available with 128GB or 256GB of ECC memory, but process dumps should not be saved unconditionally based on the server’s total memory capacity. A core file can contain passwords, request bodies, and application data. Collection scope, storage limits, and access permissions must be configured together.

1. Identify the crash type and the existing collector

After payment, find your connection credentials in the control panel and log in over SSH. The following steps require administrator privileges. Before testing, check the system, available disk space, and the current core dump handler:

cat /etc/os-release
df -h /var/lib
sysctl kernel.core_pattern
sudo journalctl -b -p warning --no-pager

When kernel.core_pattern starts with a pipe character, the kernel passes the dump to a userspace handler rather than writing it directly to the current directory. Another crash collector may already be installed. Record the original value and the relevant packages; do not configure two competing handlers and assume both will work.

First distinguish a voluntary exit from termination by a signal or an out-of-memory kill. SIGSEGV and SIGABRT typically trigger core dumps; SIGKILL does not. A service status showing an abnormal exit does not mean a downloadable file has been created.

A core dump is evidence from process memory, not an ordinary runtime log. Never place it in a public directory or attach it to a support ticket or code repository without inspecting it first.

Install the tools used in this walkthrough. If the package manager proposes replacing an existing collector, review the impact first and explicitly choose one mechanism:

sudo apt-get update
sudo apt-get install systemd-coredump gdb gcc
sysctl kernel.core_pattern
systemctl status systemd-coredump.socket --no-pager

After installation, the handler should point to systemd’s core dump processor. The socket receives incoming dumps, and the processing service starts on demand. The absence of a continuously running collector process does not mean collection is broken.

2. Set size limits for the collection chain

Create /etc/systemd/coredump.conf.d/60-controlled-capture.conf with the following configuration:

[Coredump]
Storage=external
Compress=yes
ProcessSizeMax=256M
ExternalSizeMax=256M
MaxUse=512M
KeepFree=2G

These limits are for validating a small test process, not recommended values for memory-intensive production workloads. ProcessSizeMax limits the size of dumps that can be processed, while ExternalSizeMax limits the size saved externally. If a limit is exceeded, you may have only event metadata rather than a complete memory dump.

MaxUse and KeepFree control the space used by the external dump directory. They do not guarantee reserved disk space or limit the size of the journal database. Concurrent crashes can still put pressure on storage, so also monitor free space on the underlying filesystem.

Subsequent processing instances read the configuration. Check the merged result with:

systemd-analyze cat-config systemd/coredump.conf

Before enabling collection for production, measure the memory footprint and actual dump size of a representative process, then adjust the limits. A successful test with a small program does not establish that a memory-intensive database can also be captured in full.

3. Validate a real crash with an isolated service

Build a test program without production data

Write this minimal program to /tmp/crash-lab.c:

#include <stdlib.h>

int main(void)
{
    abort();
}

Create a dedicated system account and compile the program. -g retains debugging information, and -O0 makes the test call stack easier to inspect:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin crashlab
sudo install -d -m 0755 /usr/local/libexec
sudo gcc -g -O0 -o /usr/local/libexec/crash-lab /tmp/crash-lab.c

If the account already exists, confirm its purpose rather than creating it again. Then create /etc/systemd/system/crash-lab.service:

[Unit]
Description=Isolated core dump capture test

[Service]
Type=oneshot
User=crashlab
ExecStart=/usr/local/libexec/crash-lab
LimitCORE=infinity
Restart=no

Here, infinity removes the process-level limit only for the test service. The collector remains subject to the limits in the previous section. Do not set this as a global default for all services.

sudo systemctl daemon-reload
sudo systemctl start crash-lab.service
sudo systemctl status crash-lab.service --no-pager
sudo coredumpctl --no-pager list /usr/local/libexec/crash-lab

The start command is expected to report failure. You should see a crash record matching the test executable. If no record appears, check the handler, service limits, collector logs, and disk space in that order rather than sending a crash signal to a production process.

4. Validate the dump file and debugging artifacts

Note the PID for this test from the list, enter it in the same administrator shell session, and export the dump to a dedicated private directory:

sudo -i
read -r -p 'Crash PID: ' CRASH_PID
umask 077
install -d -m 0700 /root/crash-evidence
coredumpctl info "$CRASH_PID"
coredumpctl dump "$CRASH_PID" --output=/root/crash-evidence/test.core
gdb -q -batch \
  -ex 'set pagination off' \
  -ex 'thread apply all bt' \
  /usr/local/libexec/crash-lab /root/crash-evidence/test.core
exit

First verify the event time, executable, signal, and storage status to avoid selecting the wrong record if a PID has been reused. Validation should show SIGABRT, with a call stack that can be traced to the test program’s main. Missing source code for system library frames does not prevent validation of the test chain.

Metadata alone, a file that cannot be exported, or a truncation warning does not count as successful full collection. An event appearing in coredumpctl also does not guarantee that the external file still exists.

For production workloads, archive the executable from the time of the crash, its shared libraries, and the matching debug symbols as well. An updated binary with the same name is not the original build. Use readelf -n to check the Build ID, and record package versions and build identifiers.

Do not start with bt full, which prints local variables. Inspect the basic call stack first. If you need to examine variables, work in a restricted environment and review all exported text for sensitive content. When copying binaries, also retain their hashes to guard against artifacts being replaced during analysis.

5. Enable production collection and verify rollback

To enable collection for a production service, set LimitCORE in that service’s own drop-in, run daemon-reload, and restart the service as part of a controlled change. Existing processes do not automatically receive new limits when a configuration file changes. Check the actual process limits in /proc/PID/limits.

The integration check should cover at least four points:

• Whether automatic restarts cause repeated dumps and whether directory usage remains controlled.
• Whether the limits allow a representative production process to be saved in full.
• Whether the build archive provides the matching binaries, libraries, and debug symbols.
• Whether the export directory is accessible only to authorized personnel and whether the artifacts contain sensitive data.

After testing, remove the test service file, program, and source code, then run daemon-reload and reset-failed. Handle the test dump separately using the actual path reported by coredumpctl info; do not use wildcards to empty the entire dump directory.

To withdraw the collection policy, remove the configuration file added in this walkthrough. If you need to restore the original collector, restore it through the package configuration and check kernel.core_pattern again rather than reconstructing the handler from memory. Finally, record a complete validation result: the crash event can be queried, the file can be exported, the call stack matches the correct build, and access to the artifacts is restricted. That leaves you with verifiable crash evidence, not merely an untested setting.

Frequently asked questions