Sample output:
report-02-12-2024_11-23-15.zip

I think there are two possible challenges:

• It breaks the formatting of the logs which will likely make any analysis based on complete logs from a logging stack more complicated. This is likely not a blocker as other things might break it as well (e.g. our startup logging is a bit clear as it does not inject in the middle of a log, but it still doe not have the proper format). Similarly enabled GC logging would break this. But something to keep in mind.
• It will use part of the log buffer and thus it might mean the collected logs will not contain as much information from the regular logs (for a basic Kafka cluster without any real clients this seems to do over 1000 lines in the log output - not sure how it grows). Again, I guess this is not a blocker ... it might matter a lotin some cases and not at all in others 🤷 .

As I said, probably not blockers. But it makes me wonder if this should be an optional thing enabled by some option?

IMO, the thread dump should be collected by default, not optional, since you never know if this information useful or not. How about we collect logs twice: 1 without thread dump, 2 with thread dump. Even if there are duplicated logs, it should be fine for better investigation. That should also resolve all the concerns above. WDYT?

That partially mitigates the second issue - but only partially because the thread dump will be still in the buffer consuming the space. Doesn't do anything with the first issue.

Have you instead considered some better solution that actually addresses the issues? For example something like this?

kubectl exec <PodName> -ti -- /bin/bash -c 'for proc in $(ls -1 /proc/ | grep [0-9]); do if echo "$(ls -lh /proc/$proc/exe 2>/dev/null || true)" | grep -q java; then jcmd $proc Thread.print; fi; done'

I have thought about jcmd, but it also output the thread dump in the logs as kill -3 does. Unless we're accepting to install jstack or other troubleshooting tools in pod, we might need to include them in logs. IMO, we don't need to install other tools for this, appending in logs is enough.

No, it prints it in its own stdout (i.e. the one of the kubectl exec). Also, jcmd seems to be present in the Strimzi containers.

@scholzj thanks for the jcmd suggestion. I'm using a different command with jcmd, which I think it is easier to grasp and maintain. I'm also creating a dedicated folder for thread dumps. Sample output attached.
report-03-12-2024_12-02-11.zip