So, here is a sample output from a salt-ssh command - would you expect results to replace [LEVEL] prefixes with [LEVEL] : hostname :

Is that what you have in mind?

[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Guessing ID. The id can be explicitly in set /etc/salt/minion
[INFO ] Found minion id from generate_minion_id(): salt.my.domain
[TRACE ] loading log_handlers in ['/var/cache/salt/extmods/log_handlers', '/usr/lib/python2.6/site-packages/salt/log/handlers']
[TRACE ] Skipping /var/cache/salt/extmods/log_handlers, it is not a directory
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found the in the configuration. Not loading the Logstash logging handlers module.
[DEBUG ] Configuration file path: /etc/salt/master
[TRACE ] loading roster in ['/var/cache/salt/extmods/roster', '/usr/lib/python2.6/site-packages/salt/roster']
[TRACE ] Skipping /var/cache/salt/extmods/roster, it is not a directory
[TRACE ] Added flat.compile_template to roster
[TRACE ] Added flat.get_roster_file to roster
[TRACE ] Added flat.targets to roster
[TRACE ] Added cache.extract_ipv4 to roster
[TRACE ] Added cache.targets to roster
[TRACE ] Added ansible.get_roster_file to roster
[TRACE ] Added ansible.targets to roster
[TRACE ] Added scan.targets to roster
[TRACE ] loading render in ['/var/cache/salt/extmods/renderers', '/usr/lib/python2.6/site-packages/salt/renderers']
[TRACE ] Skipping /var/cache/salt/extmods/renderers, it is not a directory
[TRACE ] Added mako.render to render
[TRACE ] Added pydsl.render to render
[TRACE ] Added stateconf.add_goal_state to render
[TRACE ] Added stateconf.add_implicit_requires to render
[TRACE ] Added stateconf.add_start_state to render
[TRACE ] Added stateconf.extract_state_confs to render
[TRACE ] Added stateconf.has_names_decls to render
[TRACE ] Added stateconf.nvlist to render
[TRACE ] Added stateconf.nvlist2 to render
[TRACE ] Added stateconf.rename_state_ids to render
[TRACE ] Added stateconf.render to render
[TRACE ] Added stateconf.rewrite_single_shorthand_state_decl to render
[TRACE ] Added stateconf.rewrite_sls_includes_excludes to render
[TRACE ] Added stateconf.state_name to render
[TRACE ] Added stateconf.statelist to render
[TRACE ] Added jinja.render to render
[TRACE ] Added gpg.decrypt_ciphertext to render
[TRACE ] Added gpg.decrypt_object to render
[TRACE ] Added gpg.render to render
[TRACE ] Added yamlex.deserialize to render
[TRACE ] Added yamlex.render to render
[TRACE ] Added yaml.get_yaml_loader to render
[TRACE ] Added yaml.load to render
[TRACE ] Added yaml.render to render
[TRACE ] Added json.render to render
[TRACE ] Added msgpack.render to render
[TRACE ] Added genshi.render to render
[TRACE ] Added cheetah.render to render
[TRACE ] Added wempy.render to render
[TRACE ] Added pyobjects.get_file_client to render
[TRACE ] Added pyobjects.load_states to render
[TRACE ] Added pyobjects.render to render
[TRACE ] Added py.render to render
[DEBUG ] Jinja search path: ['/var/cache/salt/master/files/base']
[DEBUG ] Rendered data from file: /etc/salt/roster:
testg: 10.0.0.48

[DEBUG ] Results of YAML rendering:
OrderedDict([('testg', '10.0.0.48')])
[INFO ] minions list: {'testg': {'host': '10.0.0.48'}}
[TRACE ] loading fileserver in ['/var/cache/salt/extmods/fileserver', '/usr/lib/python2.6/site-packages/salt/fileserver']
[TRACE ] Skipping /var/cache/salt/extmods/fileserver, it is not a directory
[TRACE ] Added roots.dir_list to fileserver
[TRACE ] Added roots.envs to fileserver
[TRACE ] Added roots.file_hash to fileserver
[TRACE ] Added roots.file_list to fileserver
[TRACE ] Added roots.file_list_emptydirs to fileserver
[TRACE ] Added roots.find_file to fileserver
[TRACE ] Added roots.serve_file to fileserver
[TRACE ] Added roots.symlink_list to fileserver
[TRACE ] Added roots.tagify to fileserver
[TRACE ] Added roots.update to fileserver
[TRACE ] Lockfile /var/cache/salt/master/file_lists/roots/.base.w created
[TRACE ] Lockfile /var/cache/salt/master/file_lists/roots/.base.w removed
[DEBUG ] LazyLoaded local_cache.prep_jid
[TRACE ] loading wrapper in ['/var/cache/salt/extmods/wrapper', '/usr/lib/python2.6/site-packages/salt/client/ssh/wrapper']
[TRACE ] Skipping /var/cache/salt/extmods/wrapper, it is not a directory
[TRACE ] Added state.high to wrapper
[TRACE ] Added state.highstate to wrapper
[TRACE ] Added state.low to wrapper
[TRACE ] Added state.show_highstate to wrapper
[TRACE ] Added state.show_lowstate to wrapper
[TRACE ] Added state.show_sls to wrapper
[TRACE ] Added state.show_top to wrapper
[TRACE ] Added state.sls to wrapper
[TRACE ] Added state.top to wrapper
[TRACE ] Added config.backup_mode to wrapper
[TRACE ] Added config.dot_vals to wrapper
[TRACE ] Added config.get to wrapper
[TRACE ] Added config.manage_mode to wrapper
[TRACE ] Added config.merge to wrapper
[TRACE ] Added config.option to wrapper
[TRACE ] Added config.valid_fileproto to wrapper
[TRACE ] Added pillar.data to wrapper
[TRACE ] Added pillar.get to wrapper
[TRACE ] Added pillar.item to wrapper
[TRACE ] Added pillar.items to wrapper
[TRACE ] Added pillar.raw to wrapper
[TRACE ] Added grains.filter_by to wrapper
[TRACE ] Added grains.get to wrapper
[TRACE ] Added grains.item to wrapper
[TRACE ] Added grains.items to wrapper
[TRACE ] Added grains.ls to wrapper
[DEBUG ] Executing command: ssh 10.0.0.48 -o KbdInteractiveAuthentication=no -o PasswordAuthentication=no -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o Port=22 -o IdentityFile=/etc/salt/pki/master/ssh/salt-ssh.rsa -o User=root pwd
[DEBUG ] Child Forked! PID: 27839 STDOUT_FD: 6 STDERR_FD: 8
[DEBUG ] Terminal Command: /bin/sh -c ssh 10.0.0.48 -o KbdInteractiveAuthentication=no -o PasswordAuthentication=no -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o Port=22 -o IdentityFile=/etc/salt/pki/master/ssh/salt-ssh.rsa -o User=root pwd
[DEBUG ] /root
[TRACE ] loading output in ['/var/cache/salt/extmods/output', '/usr/lib/python2.6/site-packages/salt/output']
[TRACE ] Skipping /var/cache/salt/extmods/output, it is not a directory
[TRACE ] Added compact.output to output
[TRACE ] Added no_return.output to output
[TRACE ] Added grains.output to output
[TRACE ] Added overstatestage.output to output
[TRACE ] Loaded no_out as virtual quiet
[TRACE ] Added quiet.output to output
[TRACE ] Loaded json_out as virtual json
[TRACE ] Added json.output to output
[TRACE ] Added nested.output to output
[TRACE ] Added raw.output to output
[TRACE ] Added highstate.output to output
[TRACE ] Loaded yaml_out as virtual yaml
[TRACE ] Added yaml.output to output
[TRACE ] Added virt_query.output to output
[TRACE ] Added key.output to output
[TRACE ] Added txt.output to output
[TRACE ] Added newline_values_only.output to output
[TRACE ] Loaded pprint_out as virtual pprint
[TRACE ] Added pprint.output to output
[TRACE ] data = {'testg': {'retcode': 0, 'stderr': '', 'stdout': '/root\n'}}
testg:
----------
retcode:
0
stderr:

stdout:
    /root

Yes, at least for me having [LEVEL] : hostname would make reading logs much more comfortable.

That would be useful. Thanks for the request!

Has anything been done in this department? Regardless, I have a couple of other suggestions regarding salt-ssh logging.

The base64 dump of the thin shim shows up 2 or 4 times in the debug log. Either find some way to scrub the base64 dump from the log line, or decrease the log level to trace. EDIT: Nevermind this, #27793

The salt-call command on the remote system is called with -l quiet, which means if you want to debug something, you have to SSH into the remote system, painfully reconstruct the salt-call command just to replace -l quiet with -l debug. I think the salt-call log level should inherit the local salt-ssh log level, and the remote logger output should be forwarded to the local stdout but made to stand out in some way - for example prefixing it with [REMOTE] hostname:.

I did some work to eliminate the shim printing awhile ago, and thought I had gotten rid of all of them, but I guess I need to do more looking.

The reason the command is run with log-level quiet is (if I remember correctly) because we have to parse the output of the command after running it. Getting clean returns back on the console is difficult if there are logs as well...

Anyway, thanks for bumping this, it'll get me thinking about the problem again. Because it's a real problem, debugging salt-ssh is a pain right now.

Salt-call on the client is run with --out json. If log can be part of the output and also be in json format this would simplify parsing.

@basepi Nevermid the shimming bit, I figured that out: #27793

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

Please don't close this issue.

Thank you for updating this issue. It is no longer marked as stale.

Actually it's in a funny state; salt-ssh logs:

     Function: pkg.latest
      Result: False
     Comment: The following targeted packages failed to update. See debug log for details: (salt-common, salt-minion).

except I do not seem to be able to create a debug log on the client, and the log it has created has nothing to say about this. Especially hard since doing pkg.install the same way actually works.

(In my case it's been pkg.latest vs only_upgrade=True with a not yet installed package. I am not sure how was I supposed to figure this out without trial&error.)

I use following patch to workaround this:

diff --git a/salt/client/ssh/ssh_py_shim.py b/salt/client/ssh/ssh_py_shim.py
index 5e5dbdc..7b8a156 100644
--- a/salt/client/ssh/ssh_py_shim.py
+++ b/salt/client/ssh/ssh_py_shim.py
@@ -279,6 +279,8 @@ def main(argv):  # pylint: disable=W0613
         '--metadata',
         '--out', 'json',
         '-l', 'quiet',
+        '--log-file', '/var/log/salt-ssh.log',
+        '--log-file-level', 'info',
         '-c', OPTIONS.saltdir
     ]
 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

what do you want, bot?

Thank you for updating this issue. It is no longer marked as stale.

I think this issue should be a higher priority. It should be easier to see errors output by salt-ssh. It doesn't feel like salt-ssh is supported (this issue represents a bug, not a feature).

Also, in addition to modifying the sourcecode (as documented above), you can also call it manually on the host per Debugging salt-ssh - Salt SSH:

To determine the location of salt-call, simply run salt-ssh with the -ltrace flag and look for a line containing the string, SALT_ARGV. This contains the salt-call command that salt-ssh attempted to execute.

It is recommended that one modify this command a bit by removing the -l quiet, --metadata and --output json to get a better idea of what's going on the target system.

I agree with that requirement / bug report too. It is essential to get get log output from state/execution modules running on a minion for problem analysis and supporting the salt project. Currently I'm faced with a problem caused by the pip.installed module function, which calls the python interpreter at least once as user root instead with one given by the user - option.

•    '--log-file', '/var/log/salt-ssh.log',
  

•    '--log-file-level', 'info',
  

This helped me when trying to diagnose rbenv brokenness when using salt-ssh as a non-root user. Thanks!

@kt97679 Closing this due to age, the old version of Salt and Python 2.
Can you retest this with the latest release of Salt 3006.2 and if still an issue, please open a new issue, which will have metrics in tracking issues.

Noting that with Salt 3006, having to run as root is being reduced with the user and group salt.