[RFC] Sixel support

[ctrlcctrlv]

Sixels are an ancient protocol for supporting graphics in the terminal, similar, but inferior to, Kitty's native graphics protocol.

The major way sixels are inferior is that there is no alpha and the color palette is limited, unlike Kitty's support for full RGBA.

Be that as it may, due to their age, many other terminals support them, which becomes a self-fulfilling prophecy, as people are more likely to add sixel support than Kitty graphics support.

So, I wonder what the right play is here. Implement sixels? Not hard to do, just translate the commands into Kitty graphics commands internally.

Or, given our market leader position, ignore sixels, and continue to ask other VTEs to implement our protocol?

[kovidgoyal]

On Sun, Apr 05, 2020 at 11:58:55AM -0700, Fredrick Brennan wrote: [Sixels](https://en.wikipedia.org/wiki/Sixel) are an ancient protocol for supporting graphics in the terminal, similar, but inferior to, Kitty's native graphics protocol. The major way sixels are inferior is that there is no alpha and the color palette is limited, unlike Kitty's support for full RGBA. Be that as it may, due to their age, many other terminals support them, which becomes a self-fulfilling prophecy, as people are more likely to add sixel support than Kitty graphics support. So, I wonder what the right play is here. Implement sixels? Not hard to do, just translate the commands into Kitty graphics commands internally.

sixels have other problems, no z-index, much harder to implement, no support for efficient data transmission when the emulator and program have a shared filesystem/memory. See the original issue for graphics support in kitty (it is linked to from the graphics protocol page) for a discussion.

Or, given our market leader position, ignore sixels, and continue to ask other VTEs to implement our protocol?

As I noted on the original issue, I have no interest personally in implementing an inferior graphics protocol in kitty. I know of no program that I would use that supports sixels but not kitty's graphics protocol. That said, I wont refuse a patch to implement it on top of the existing graphics support, provided I was reasonabl confident the person implementing it would stick around to maintain it as well. Although I suspect it wont be quite trivial, as the semantics of how images drawn with sixels work with scrolling, screen boundaries, cursor position and text output are probably different.

[kovidgoyal]

For reference: #33

[kovidgoyal]

Since no one else has comments, closing.

[ctrlcctrlv]

Sixel's are once again on my mind. They're supported in a lot of programs, including mpv (Cf. #3673, saitoha/libsixel README.md).

I'd appreciate this being reopened, but even if it's not, I'm going to spend tonight integrating libsixel as an optional build dependency into Kitty. It's a C library with a Python binding. I think I can do it in a few hours.

[kovidgoyal]

The hard part isnt parsing the escape codes, which libsixel will do for you, its adding support for the very different semantics that sixels based images have wrt to things like scrolling, text overwriting the image, what happens when the terminal resizes, etc.

If I were you I would instead spend this energy on adding support for the kitty graphics protocol to mpv :)

[ctrlcctrlv]

I have the basics of Sixel support working.

I'm using xterm -ti vt340 as the source of truth for what should happen in various situations. I feel like xterm is already a logical source of truth anyway.

I posted the code to try to answer some of @dankamongmen's questions. No, sixel images don't blow away Kitty graphics data. They work together, they're implemented through grman_handle_command!

[kovidgoyal]

Good progress, now try the following things:

1. Sending an image larger than the screen
2. Moving the cursor back and printing some text over the image

See if what happens matches xterm.

[ctrlcctrlv]

Test case 1

I fixed this by switching from grman_handle_command to screen_handle_graphics_command.

Test case 2

I'm still working on this one, as you can see it's quite different. Any ideas?

I generated this test case by appending this to the .sixel file…

\x1b[10Ahello\x0a

[ctrlcctrlv]

Actually after I adjusted the z-index to -1, it's just a matter of fixing the placement of "hello". I think that the white background of the cells in xterm should be optional, but if it's really important to you I can fix that too.

New test case 2

[kovidgoyal]

Not sure I am following about the white background. I think the sixel semantics are that both the foreground and background colors are displayed for text printed over an image. @dankamongmen can correct me if I am wrong. For you I think that means the z-index needs to be low enough that the image is displayed below the background layer (IIRC this was a feature of the graphics protocol you requested :)

It is important to have the cursor positioned at the same location post drawing a sixel as xterm does it.

[ctrlcctrlv]

Got it. -2 is correct.

I've discovered that xterm considers a Sixel sequence as having an implied newline, even if the file is noeol. If the sequence is followed by a newline, however, that's not an extra newline.