add support for some signed texture formats

[oltolm]

Even though we can not support the general case, we can easily support the case when all components are signed. See #587.

[oltolm]

@abaire Sorry for the ping, but can you take a look?

[mborgerson]

FYI while glancing at @coldhex's development branch in regards to #1895 and #1896, I noticed they also had been working on signed texture support. I haven't reviewed their code yet, but we should coordinate here to avoid duplicate efforts.

[oltolm]

Sure, I am all for it.

[oltolm]

I think this PR is useful regardless of other PRs because it solves the problem in the specific case when all components are signed. It does not conflict with other more general solutions and can coexist with them.

[oltolm]

@coldhex Can you take a look?

[coldhex]

R might usually be unsigned. See also the texture data conversion done in, which should change depending on if signs are set or not:

xemu/hw/xbox/nv2a/pgraph/texture.c

Lines 380 to 397 in a008207

	} else if (s.color_format == NV097_SET_TEXTURE_FORMAT_COLOR_SZ_R6G5B5) {
	assert(depth == 1); /* FIXME */
	size = width * height * 3;
	converted_data = g_malloc(size);
	for (int y = 0; y < height; y++) {
	for (int x = 0; x < width; x++) {
	uint16_t rgb655 = *(uint16_t *)(data + y * row_pitch + x * 2);
	int8_t *pixel = (int8_t *)&converted_data[(y * width + x) * 3];
	/* Maps 5 bit G and B signed value range to 8 bit
	* signed values. R is probably unsigned.
	*/
	rgb655 ^= (1 << 9) | (1 << 4);
	pixel[0] = ((rgb655 & 0xFC00) >> 10) * 0x7F / 0x3F;
	pixel[1] = ((rgb655 & 0x03E0) >> 5) * 0xFF / 0x1F - 0x80;
	pixel[2] = (rgb655 & 0x001F) * 0xFF / 0x1F - 0x80;
	}
	}
	} else {

[oltolm]

I don't understand what this code does, I can't do it.

[coldhex]

Doesn't this only test if one of the sign bits match, not all?

[oltolm]

Yes, I fixed it.

[coldhex]

I also considered a solution similar to this PR, but then went with implementing a general solution in my dev branch. While this PR should improve water effect in e.g. Halo 2, I have no idea if it might introduce artifacts in other games. To be safe, even if B8G8 or R8B8 texture are used, all sign bits R, G, B, A should be set if SNORM texture is used (because for e.g. B8G8 texture, hardware uses data from B and G channels also for R and A channels, but the sign setting is independent for each channel.) However, requiring all channels RGBA to have the sign bits wouldn't fix Halo 2 water, because Halo 2 doesn't set them all, and therefore is really not an option.

(And as far as my dev branch solution is concerced, it works fine, but I am still considering implementing signed texture handling in a different way. At some point I'm going to try a solution that does bilinear (or trilinear in case of mip-mapping) texture filtering manually in the fragment shader. This would avoid the need to adjust texture channel data sign bits before uploading to GPU.)

[oltolm]

I don't know if makes any game worse, but it fixes the water in Ninja Gaiden 2.