What do you expect to happen when running that command inside a pane that does not cover the entire terminal window? Even if you have a single pane (which is my typical tmux window) do you expect, or not, that the image will cover the tmux status line? The issue here is that tmux would have to know about the SIXEL protocol in order to correctly map coordinates if it were to restrict the image to a pane. And that isn't a reasonable requirement for a feature that 99.9999% of users don't care about.

"99.9999%" seems like an inflated figure if we go by the number of stars on the libsixel project. And anyway, I'm not really expecting it to intelligently map coordinates -- I would just like it not to eat the SIXEL data stream like it seems to be doing by default. If it draws overtop of existing windows, presumably I can just do a refresh. It's a bit of an ugly hack, but like you said, it would be difficult to fully support SIXEL.

Edit: also, it's worth noting that if this feature was better supported by common tools, maybe more people would use it and become invested in it. "No one uses it so we're not going to support it" sounds a bit like a self-fulfilling prophecy to me -- after all, isn't the value of a tool often dictated by its interoperability?

To expand on this, I am not going to work on this but I am not rejecting the idea. If someone is going to come up with a patch then I will advise and review it. It needs to work properly, by which I mean:

• no external dependencies;
• SIXEL images need to be saved and redrawn in the right place if needed;
• they need to be properly clipped at the pane borders.

The existing passthrough escape sequence should already be adequate for using SIXEL without tmux knowing about it.

SIXEL support in tmux would be very much appreciated!

I run tmux on all my terminals, so if tmux can't handle SIXEL, that means I can't use it. Currently I have to resort to drawing Unicode half-blocks, but it's not pretty.

SIXEL might be old and crufty, but it's THE standard for graphics drawing in the terminal. If more terminals and related software adopted it, that would be great indeed.

@nicm "no external dependencies" doesn't seem all that feasible, but what about optional dependencies -- have SIXEL support enableable during compilation time -- at which point it requires libsixel. That way no one who installs tmux, but doesn't use SIXEL, is obligated to install it.

If you write it using libsixel then an optional dependency would probably do fine. But there is no chance of adding libsixel to OpenBSD so it would need to have minimal changes to the main tmux code (so all be wrapped up in a separate file with only a few lines of #ifdef in the main codebase).

Patch to allow raw sixel through (similar to the passthrough escape sequence): tmux-raw-sixel.diff.txt

• Bump the sequence timeout to 5 seconds instead of 100 ms for more time for SIXEL images but still not let the terminal hang forever. This could be made more sophisticated, it sort of depends how applications actually generate SIXEL in practice.

• Check the code in DA for SIXEL support and also check a terminfo(5) extension flag Sxl (this can be added in terminal-overrides for terminals that do not set DA properly).

• Recognise SIXEL sequences and copy them to the attached outside terminals if they support SIXEL (DA or Sxl set).

The SIXEL image will disappear as soon as tmux redraws, which will be whenever it is necessary to keep the terminal up to date, same as using the passthrough escape sequence already.

I am told that the slow terminal redraw code also interferes with SIXEL images but I suspect this is only with large images and I don't have any at the moment.

An issue with full support is how to handle mapping images to cells so we can crop the image to the pane correctly.

If we have a SIXEL image of say 64x64 pixels and terminal A where each cell is 8x16 and terminal B where each is 16x32, then on A the image takes up 8x4 cells but on B only 4x2. So if image is drawn as given on both terminals it will look different relative to the surrounding text.

I think what we would need to do is work out the "best" cell size for each pane - perhaps the largest from all of the attached clients, or some sensible default if unattached - and set that with TIOCSWINSZ. This would mean applications inside tmux would always be generating their images for the "best" cell size and in the case of one attached client (or all clients with the same font size), the images would be correct most of the time.

We would then need to record the cell size with the image and use that to work out how to crop the image needed. So if an image is 64x64 and was written when the cell size was 8x16 and we need to draw only its leftmost 3 cells we would know we need to crop it to 24x64.

Then when we draw the image (or part of it) we would need to compare the cell size when the image was written with the cell size of the target terminal and if they differ, scale the image. So if we had the same image received while attached to terminal A and we now want to write it to terminal B, we need to double its size so it is 128x128 and takes up 8x4 cells on terminal B like it originally did on terminal A.

WIP with basic APIs for parsing/scaling/generating SIXEL. Doesn't do anything at the moment, the hard part (making tmux store them and display them in the right place) is still to do: tmux-sixel.diff.txt

Dropping by, as I saw your HN comment. You mentioned you need more sixel examples so I forked the sixel branch on https://github.com/csdvrx/tmux where I added sixels/ with the traditional sixel test images and a gnuplot script that you can use with X Y parameters to generate test images of any size. This should allow you to check how the slow terminal code interacts.

Having sixel pictures disappear when changing panes is already good. If you want to do more, there seems to be 2 ways:

1. if you want to be able to redraw, you want to keep the images around: mlterm and other simply save the .six files. Storing sixels create the problem of where to save (suggestion: ~/.tmux/%PID%PANE) and how much to keep (suggestion: as many as would fit given the history size and screen resolution, with each sixel converted to an equivalent height in lines: vimperator3 is almost a full page on my terminal).

2. You could also just keep the full string in memory. Sixels are not large. A simple rule of thumb could be 5 sixels per panel. Or just 1.

Still I am not sure keeping many pictures is such a good idea. There is little interest in displaying sixels from the scroll history. They are mostly useful for the live commands. Instead, I would suggest using libcaca for off screen images.

3. the alternatively is that all sixel images would be converted to ANSI art and stored in the history as such. Then, upon redraw, tmux could display this close match of the sixel image. It would also provide something to be shown on terminals that do not provide sixel support, a fallback mode (more accurate with 16 million colors, but better than nothing even with 256 colors). It would also remove the resizing problem: just show the sixel picture on the tmux that generated it (as you already have the cell size), and show the ansi art of a matching size on others.

The license of libcaca is very permissive, so it should not be a problem to have it as a dependency. It would also allow a simpler way to start, with a solution that could later be refined based on needs (ex: first use the best cell size and scale to show on other connected tmux) and later figure the best way to display sixel when scrolling through history

What do you think?

I think the best approach would probably be to have a global limit on either the total number or total size of SIXELs in the server. When a new one is generated, the oldest or least recently drawn is thrown out.

Saving images and indeed any pane content to disk without the user explicitly saying it is OK opens up security questions I would rather not have.

I don't think a dependency on libcaca or similar is necessary, it wouldn't be hard for tmux to draw an ASCII representation itself, or a plain coloured box would be enough for missing images. The idea would be just to show that SOMETHING was there, if the user wanted to see it they should use a SIXEL terminal or configure tmux to keep more images.

The biggest complexity is not really storing them anyway, it is knowing when to redraw them, scroll them, remove them. I think a good first step would be to: 1) just redraw the whole pane anytime it changed with a SIXEL visible (this would be slow but fast enough with a local terminal - we already do this for terminals which don't support scroll regions or margins or BCE) 2) remove SIXELs or convert to ASCII anytime they are damaged (a cell underneath them is modified or they are split into two parts by an insert or delete) 3) likewise remove them when they scroll into the history, I think it is important to support them scrolling them when they are visible.

Thanks for the test images, I'll take a look.

I like the idea of converting to ASCII anytime dealing with an image becomes too difficult (it is overwritten or broken up or we have too many images) - it keeps things simple while still letting the user know something was there.

Well, I made a quick test to convert it to ASCII and store it in the grid so that when it redraws and the image vanishes you get that but there are a couple of problems:

• Putting it in the grid means it gets wrapped which is different from how images behave and pretty ugly - they should be truncated. So we would need a line flag to not wrap, and work out how to turn it off when needed.

• There are no good terminals! xterm won't display SIXELs properly half the time. mlterm usually will, but it doesn't seem to support 256 colours let alone RGB. I got it to work! :-)

Not sure about this, might be better to treat images as an overlay.

It would be cool to have SIXEL support in tmux.
For Windows users: mintty can display SIXEL fairly well.

Complete support is a lot of work I don't have time to work on this any further right now, the work I have done so far is in the sixel branch but there is a lot more to do. It works OK in that it will clip SIXEL images to the containing pane, but they will still be overwritten by other text and removed on redraw.

I have also added a branch sixel-passthrough which passes SIXEL images through completely unmodified, obviously images will overwrite panes and the status line, and if you change pane or resize or otherwise cause tmux to redraw the image will disappear.

There is still one problem I know of where if tmux needs to redraw for another reason while the SIXEL image is being written it will be removed as soon as it has appeared, I'm not sure there is actually anything we can do about that.

Still, one of these should be enough for basic use if you want to try them.

For reference, here is what needs to be done for full support:

• Incoming SIXEL images need to be stored so they can be redrawn, with a limit on either size or number of images. (easy)

• The redraw code needs to work out what SIXEL images are visible when redrawing a pane or window and send them as well. (not too hard)

• When scrolling a pane, SIXEL images need to be moved up as well. They need to be removed when they drop out of the history, or perhaps when they go completely off the visible screen instead. (quite hard)

• The cursor needs to move correctly to below the SIXEL image when it is first sent, including scrolling if necessary. (quite hard)

• Something needs to be done with SIXEL images when they are overwritten or broken up - this needs to be detected and then the image needs to be removed, converted to ASCII, or whatever. (quite hard)

• What happens if the terminal or network is slow? If we send a SIXEL image and by the time it is finished we need to redraw again, perhaps because the application in the pane has updated in a way that needs a full redraw, what is to stop this becoming an infinite loop of redrawing? Perhaps we should disable SIXEL if the terminal takes too long to consume an image? (not sure what to do here)

There are probably more things I haven't thought of.

About redrawing and scrolling, ideally sixels that are present on the displayed pane should be visible when returning to that pane - for all open panes. But when sixels move out to the scroll history, they are not as important anymore.

If you want to keep the exact sixels, saving them in a fifo in memory in a fifo seems fine: if they are available, show them, if not, show their libcaca ansi art that fit in the scroll history. But I don't think there is any need to keep a perfect sixel representation in history, as it opens the can of worms of how munch to store, and where.

I haven't yet tested your patch converting sixel to ASCII but I believe libcaca ANSI art could be much better than a placeholder. This is because sixels are not just eye candy: they can also be functional and important. For me, sixels of gnuplots are a way to remotely monitor database performance. ANSI art of gnuplots would add the feature of scrolling back to see past performance of the system.

For sixels on their way out of the visible display, yes it would be nice to keep them while a part is still visible. A size of 1 displayed pane could be kept in memory: it would allow a limited scrolling before expiring the sixels. Then, like the broken / damaged sixels (ex: overwritten by ansi ), they could be replaced by a libcaca rendition. It would be simpler while still providing good functionality.

Alternatively, it may be a quick hack, but what about using the host terminal scrollback buffer? Keep a local copy of the scrollback buffer to have the full sixel inside, when the pane is focused, rewrite the whole thing, and let the host terminal handle scrolling/cutting the sixel. This would remove most of the "quite hard" points. They could be done later. The only remaining issue would be to estimate the heigh of the sixel in line, to know when it should be removed of the scrollback buffer and replaced by a text rendition.

If you are looking for terminals to test, besides xterm I also suggest mlterm (multiplatform) and mintty (windows). Lately I've become more of a windows user than a linux/bsd user!

Here is a screenshot of 2 panes tmux inside mlterm on windows.

My mintty has an interesting bug where anything on one pane affects the other (so sixels, but also ctrl-l, reset, ansi, scrolling up etc.). I believe it is due to the terminfo I wrote.

In mlterm, the problem of the sixel being erased immediately after display due to tmux having to redraw shows up sometimes when displaying too large sixels, like vimperatrix3 on one side.

By the way, you may want to reopen this issue, to collect feedback, bugreports and patches about the current sixel branch.

Well, they might not be as useful to you when they are scrolled into the history but that is not to say they aren't useful to others. But I don't think it is terribly important if they go away when not visible myself although it would be nice if they did not.

I don't want to add a dependency just to display SIXEL images on terminals which don't support them or when the user has many of them, it isn't worth it. If someone is running a program where SIXEL images are critical, I think it is reasonable to expect them to use a terminal which supports them and not rely on tmux rendering them as ASCII. Remember that only a couple of programs actually make use of these at all.

I don't know what you are suggesting with the terminal scrollback, tmux has no control over it.

If people test the sixel-passthrough branch and verify it works no worse than existing solutions using the Ptmux escape sequence or modifying the code to allow them through in other ways, then I will probably be able to merge it.

You can try the sixel branch instead if you like but it is a long way from complete and I don't think I'll be able to look at it again for a few weeks.

Properly keeping the sixels even if they go away in the scroll history would be better, for sure.

For the scrollback, I was thinking of abusing the alternate screen, like when using smcup@:rmcup@ : basically, empty the screen and write everything including sixels that may or may not fit, then let the terminal decide what is in focus and what is not, then what to do with sixels that don't fit. It is a bit of a hack.

I was also thinking of alternative ways, including a sixel buffer for each pane, as big as the current resolution, that is updated when new things are displayed. A clipped view of the buffer could be passed on redraws and pane switch. But it seems a little complicated.

I have played with your textbox feature b1904c9 and I like it a lot! I could see that filled by ansi art :-) libcaca didn't strike me as a heavy dependancy, or a license constrained dependency. It seems like the easiest way out. I will see what I can do with it.

I have tried the sixel branch and it works quite well. I would only suggest reverting 146ee3f to get placeholders, and moving the cursor to the bottom of the placeholder textbox.

I find no regression, but you may want more people to test it.

https://github.com/mintty/mintty/wiki/CtrlSeqs#image-support would be relatively easy to add at the same time once cropping is supported.

CtrlSeqs would be useful to have tmux -CC work the same in windows as in iTerm, but my main focus at the moment is for sixels.

This is because sixels are more universally supported by applications (w3m, gnuplot, mikutter...), so having a fallback for the terminals that don't support them would let anyone use these applications. Allowing a widespread use of, say w3m graphical browsing by simply using the right font + tmux, seems like a good way to spread text mode apps. If the users really like it, it would also give a good reason to change the terminal to get somethig even better.

So I did test many things. Here is a quick comparison of sixel rendering vs unicode half blocks in 256 colors, using public domain code that only use the half block ▄ with varying background and foreground (so resolution=cell size X, 2*cell size Y).

This result in 19x32 characters is actually much nicer than anything I could get of libcaca. I think it would make sense to show that in the placeholder. I'm exploring options to further increase the resolution to 2X, 2Y by using ▄▄:▗▄|▄▖|▗▖|▄ | ▄| ▗|▖ |▗ | ▖| ; and eventually 8Y, 8Y by adding to the mix; █:▁|▂|▃|▄|▅|▆|▇|█ ; █:▉|▊|▋|▌|▍|▎|▏

This higher resolution, especially in 16m colors, should provide a very graceful fackback mode.

I think something like this will be very useful because honestly SIXEL is slow to handle and support in terminals is quite buggy. I think it may be necessary to only allow one or at most two images visible at one time or to limit the maximum size in bytes of a single image. Also at the moment I have images simply disappear if they move into the history or are otherwise modified and I'm not wild about that.

The most reliable course may be to display the image at full resolution when it is first displayed and so long as it remains entirely unmodified and visible but as soon as it is scrolled even partly into history or modified in any way it becomes ASCII and thereafter is treated as if it had been written as ASCII all along.

I believe it was what I suggested before :-)

So yes, I really think it is the most reliable course of action for now!

Just show all the sixels that fit on the current pane (while saving their unicode equivalents for later use), then replace the sixels by the unicode equivalents as soon as it is required (by changing panes, scrolling in history, etc). That's version 1.

Later, when this already works well, which features make sense/are needed may become more obvious. I could see some reasons to keep all the sixels currently visible in each pane, if only to reduce the confusion when switching between panes. That would be version 2. But that's about it. I confess I have a limited imagination, and that some people may see how it would make sense for them to have their history display the cropped sixels. But I just can't imagine why for now. In the case of w3m, sixels will be redrawn when scrolling on the page.

And even in the case that sixel history feature would make sense, it's hard to guess how important it will be, and how much it may be needed - especially if the unicode is a good enough approximation.

They are too slow to be displaying them all the time, getting rid of them as soon as possible is best.

Then this is what version 1 would be: get rid of the sixels on the first redraw (ie as soon as possible), by using the unicode graphics saved instead.

I'm sending you the source code - it's from Justine Tunney, in the public domain. You should remove screen_write_box in sixel.c : sixel_to_screen() and just use grid_view_set_cell with the unicode output made by the character dimensions screen_size_x(image), screen_size_y(image) calculated in screen-write.c : screen_write_sixelimage() - this would allow you to gain 1 character in X and 1 in Y (so 2 half blocks in Y) , and use the extra resolution to make a picture closer to the sixel version, as displayed above.

So here is a use case for keeping old sixel images in the scoll buffer: I am currently trying to replace a Jupyter notebook that uses matplotlib to show plots by some python scripts that simply display their output using matplotlib-sixel. Some scripts generate several plots at once because it is more efficient, and just dump everything to the terminal. So if I want to see all those plots, relying on the scroll buffer is my only option.

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.