libfreenect2 xyz computation chain vs windows sdk v2 xyz computation chain

[timprepscius]

git log -1 --oneline
6236635 docs: Fix a typo

So I've been studying the XYZ computation chain of windows and libfreenect2.
Also the depth values generated by this chain.

windows depth:

libfreenect depth using opencl (not clkde) NOT undistorted

libfreenect after undistortion:

The depth value for the center pixel.
ms -0.002749, -0.012855, 0.645
lf2 using ms table -0.00283028,-0.0132361,0.664106 (distorted)
lf2 -0.0815203,0.0937013,0.661913 (undistorted....)

So my question is:

Does anyone know how they are computing the DepthToCameraSpaceTable?

They say they are including the distortion in the table it self.
Has anyone had any experience with creating this table?

I understand that the distortion is going to come from the distortion formulas.
https://en.wikipedia.org/wiki/Distortion_(optics)

As I understand it, libfreenect2 first undistorts the image. And then multiplies this undistorted image by a table value (X,Y) to get the XYZ.
As I understand it, MS includes the distortion in the table, and multiples the undistorted depth value by the table XY to get the XYZ.

I'm wondering if anyone can give me any clues to replicating this table.

[floe]

The original discussion (I think) can be found here: #144 (comment)

[timprepscius]

I believe I have a good start on this problem. However the data is not exact which annoys me ;-)

In a nutshell:

void RegistrationImpl::generateDepthToCameraSpaceTable (DepthToCameraSpaceTableEntry *t, int width, int height)
{
  for (int y=0; y<height; ++y)
  {
    for (int x=0; x<width; ++x)
    {
      float nx, ny;
      imageToNormalizedCoordinate((float)x, (float)y, nx, ny);

      float ux = 0, uy = 0;
      undistort(nx, ny, ux, uy);

      t->x = ux;
      t->y = uy;
      ++t;
    }
  }
}

which is very close to:
#144 (comment)

The values generated are slightly off the MS. And there is an oscillation to how off they are off.

import json
import numpy as np

l = json.load(open("mac-table-jun14747.txt"));
r = json.load(open("table3.txt"));

bins = []
for i in range(512):
        bins.append([])

i = 0
z = zip(l, r)
for t in z:
        lv, rv = t
        lv[1] = -lv[1]
        nl = np.asarray(lv)
        nr = np.asarray(rv)

        diff = nr - nl
        bin = i % 512
        bins[bin].append(diff)
        i = i + 1


i = 0
for bin in bins:
        print(" --------------------------------- " + str(i))
        for d in bin:
                print(d)

        i = i + 1

which reminds me very much of:
#144 (comment)

table3.txt

mac-table-jun14747.txt

[timprepscius]

Ahh, kinect_undistort makes perfectly aligning table numbers.. Ahhhhhhh.. I am now listening to "the girl from ipanema" in my head.

[xlz]

As I understand it, libfreenect2 first undistorts the image. And then multiplies this undistorted image by a table value (X,Y) to get the XYZ.
As I understand it, MS includes the distortion in the table, and multiples the undistorted depth value by the table XY to get the XYZ.

No, libfreenect2 does it the same way as MS.

The purpose of the precomputed "tables" is partly explained here https://github.com/OpenKinect/libfreenect2/blob/master/src/libfreenect2.cpp#L89. The comments in #144 were only investigative.

The values generated are slightly off the MS
kinect_undistort makes perfectly aligning table numbers

What values? I'm not sure what you mean here, but the actual used undistort function is indeed slightly different from MS version, said here:

I used a different implementation of kinect_undistort with Newton's method and double precision to derive the inverse of distortion.

The implementation is here https://github.com/OpenKinect/libfreenect2/blob/master/src/libfreenect2.cpp#L153. The purposes were partly to get better numeric precision and partly because the original kinect_undistort didn't make full sense to me in terms of algorithm.

[timprepscius]

Yeah. As I understand it, the libfreenect2 actually does the depth calculation differently than MS.
Perhaps I have misunderstood the code.

MS seems to do:

1. Send data to shaders.
   a. Transformations take place, in your github notes there are comments about z undistortion.
2. For each X,Y in the output depth data, multiply by a value from the depthToCameraSpace table.
   b. This depthToCameraSpace table includes the undistortion of the x-y plane.

LF2 seems to do:

1. Send data to shaders.
   a. Transformations take place, in your github notes there are comments about z undistortion.
2. Undistort resulting depth data.
   a. This rearranges pixels, and actually throws some pixels away.
3. Multiply Undistored depth data by a projection table (I forget name of table).

To generate the perfectly matching MS projection (with undistort included), you can use the kinect_undistort function, found somewhere in these github notes. Or if you wish to use the newton etc, you can user the lf2 undistort function.

[xlz]

You are basically correct. We use a different kinect2_undistort() with Newton's method etc because it is numerically superior and algorithmically equivalent.

[xlz]

I think I'll add some note to the documentation about the difference.