is it possible to use regressed 2d vertices to optimize the 3d vertices

[YongtaoGe]

Hi, @isarandi.

The NLF model can output both 2d points and 3d points from a canonical template.

My question is: Are the 2d points more accurate than the regressed 3d points?

If the answer is yes, and the camera's intrinsic and extrinsic of the image are known, can the regressed 2D points be used to optimize the 3D points?

[YongtaoGe]

I also find a strange Phenomenon by visualizing the estimated 2d points, the prediction have strange truncations on foot

[isarandi]

The image definitely looks wrong, the expected result is much better. What code are you using to produce it?
Are you taking the raw 2D heatmap results? That is expected to be a partial result, only for the inner part of the crop. The CNN has a stride of 32, so the edge of the image crop cannot be voted for in this 2D heatmap. There is a half-stride border where no prediction will appear. But that's fine, this is not the final output. It gets combined with the 3D heatmap's result and the border areas take their estimate from that.

My question is: Are the 2d points more accurate than the regressed 3d points?

The 2D points tend to be somewhat more pixel-precise, but not by a huge amount.

If the answer is yes, and the camera's intrinsic and extrinsic of the image are known, can the regressed 2D points be used to optimize the 3D points?

This already happens implicitly, since the final output is a blend of the back-projected 2D points and the translated 3D points.

[YongtaoGe]

@isarandi I take the raw 2D heatmap results to do 2d visualization.