This is probably happening because the default systemd KillMode is control-group. I wonder if setting it to mixed in the stage1 unitfile would be the correct approach.

Possibly. But they would receive SIGTERM anyway, as when the whole pod goes down (ie. systemd-nspawn killed) systemd-pid1 will do a SIGTERM+SIGKILL round anyway. I don't see any easy way out of this, except for keeping the pod running á la rkt-app.

This issue has come up again for me. There's gotta be some kind of solution or workaround. I'm a big boy, I can handle my own signals, thankyouverymuch systemd. I really don't want to use Docker for my current problem. Or worse, not run in a container at all!

There's a generated systemd .service file in the stage1 rootfs that is (or appears to be) specifically for the single application that's been spawned:

2z [root@elasticsearch-0ed6f9c740fd9491c:/proc/6933/cwd/stage1/rootfs] # cat ./usr/lib64/systemd/system/elasticsearch.service
[Unit]
OnFailure=halt.target
Description=Application=elasticsearch Image=s3.amazonaws.com/example/apps/elasticsearch
DefaultDependencies=false
Wants=reaper-elasticsearch.service
Requires=sysusers.service
After=sysusers.service
Requires=prepare-app@-opt-stage2-elasticsearch-rootfs.service
After=prepare-app@-opt-stage2-elasticsearch-rootfs.service

[Service]
Restart=no
SyslogIdentifier=elasticsearch
StandardInput=null
StandardOutput=journal+console
StandardError=journal+console
TimeoutStartSec=0
ExecStart="/usr/local/bin/launch-elasticsearch.sh"
RootDirectory=/opt/stage2/elasticsearch/rootfs
WorkingDirectory=/
EnvironmentFile=/rkt/env/elasticsearch
User=0
Group=0
NoNewPrivileges=false
MemoryLimit=26000000000
CPUQuota=650%

Why can't that single service have KillMode set to mixed or process without impacting the pod as a whole? I'm sorry, I've never used rkt with multiple apps in a pod, so perhaps I'm missing a detail (or something bigger). I need time to orchestrate the shutdown of the application, but obviously more serious action needs to be taken if the timeout's exceeded.

@blalor is this a pod with a single app inside? My understanding is that your root problem comes from the pod also being torn down with this single application.

Yes, it's a single-application pod. I need the application to shut down in a controlled fashion, whereby the process launched by systemd is able to initiate cleanup and then notify child processes to exit. I can't do that if the child processes are told to terminate by systemd (and I'm unable to inhibit the child processes' SIGTERM handling).

This is the second time I've run into this problem, which revolves around managing data for a stateful application. I'm currently attempting to get an Elasticsearch node to remove itself from the cluster by deallocating shards on shutdown. A wrapper script is responsible for starting and stopping the main ES process; when SIGTERM is received, it updates the cluster state to move shards away from the terminating node, waits for the shards to be relocated, and then sends SIGTERM to the main ES process.

I'm not sure if this works, but you may as well try: instead of stopping the pod, just enter the running stage1 (via nsenter on the systemd-pid1 process) and do a systemctl kill or a plain kill on the parent service. This should let the parent do whatever it needs to handle children and then exit. I understand this is quite dirty, but it is actually how we are planning to implement rkt signal for #1496.

I'm working with containers scheduled via Nomad; that's not a viable production solution. I can test it on Monday against a running pod if you're looking for verification of the final config of the unit, but that's not something I can entertain in a production scenario.

I started working on custom KillModes on #3732