Imgtransform extrinsics

[Serafadam]

Purpose

Adds extrinsics field to ImgTransform which is populated automatically in Camera node & Stereo node

Specification

Updates in firmware regarding population
Adding two new examples that can be used to test extrinsics of stereo node

Dependencies & Potential Impact

None / not applicable

Deployment Plan

None / not applicable

Testing & Validation

None / not applicable

[cursor]

Bug: Data Loss and Incorrect Initialization

The deserializeImgTransformation function reads extrinsics from the protobuf message, but the serializeImgTransformation function does not write them, leading to data loss during serialization. Additionally, the deserialization code accesses extrinsics fields directly without checking for their presence, which can cause issues with older protobuf messages. A separate issue is that the ImgTransformation constructor is incorrectly initialized with srcwidth and srcheight instead of width and height, resulting in incorrect dimensions.

src/utility/ProtoSerialize.cpp#L88-L125

depthai-core/src/utility/ProtoSerialize.cpp

Lines 88 to 125 in 21a0706

	std::array<std::array<float, 3>, 3> transformationMatrix;
	std::array<std::array<float, 3>, 3> sourceIntrinsicMatrix;
	std::vector<float> distortionCoefficients;
	distortionCoefficients.reserve(imgTransformation.distortioncoefficients().values_size());
	for(auto i = 0U; i < 3; ++i)
	for(auto j = 0U; j < 3; ++j) transformationMatrix[i][j] = imgTransformation.transformationmatrix().arrays(i).values(j);
	for(auto i = 0U; i < 3; ++i)
	for(auto j = 0U; j < 3; ++j) sourceIntrinsicMatrix[i][j] = imgTransformation.sourceintrinsicmatrix().arrays(i).values(j);
	for(auto i = 0; i < imgTransformation.distortioncoefficients().values_size(); ++i)
	distortionCoefficients.push_back(imgTransformation.distortioncoefficients().values(i));
	ImgTransformation transformation;
	Extrinsics extrinsics;
	extrinsics.toCameraSocket = static_cast<dai::CameraBoardSocket>(imgTransformation.extrinsics().tocamerasocket());
	extrinsics.rotationMatrix.resize(3, std::vector<float>(3));
	for(auto i = 0U; i < 3; ++i) {
	for(auto j = 0U; j < 3; ++j) {
	extrinsics.rotationMatrix[i][j] = imgTransformation.extrinsics().rotationmatrix().arrays(i).values(j);
	}
	}
	extrinsics.translation.x = imgTransformation.extrinsics().translation().x();
	extrinsics.translation.y = imgTransformation.extrinsics().translation().y();
	extrinsics.translation.z = imgTransformation.extrinsics().translation().z();
	extrinsics.specTranslation.x = imgTransformation.extrinsics().spectranslation().x();
	extrinsics.specTranslation.y = imgTransformation.extrinsics().spectranslation().y();
	extrinsics.specTranslation.z = imgTransformation.extrinsics().spectranslation().z();
	transformation = ImgTransformation(imgTransformation.srcwidth(),
	imgTransformation.srcheight(),
	sourceIntrinsicMatrix,
	static_cast<CameraModel>(imgTransformation.distortionmodel()),
	distortionCoefficients,
	extrinsics);
	if(transformation.isValid()) {
	transformation.addTransformation(transformationMatrix);
	transformation.addCrop(0, 0, imgTransformation.width(), imgTransformation.height());
	}
	return transformation;
	}

Fix in Cursor • Fix in Web

---

Bug: Calibration Error: Incorrect Socket Reference

The CalibrationHandler::getExtrinsicsToLowestSocket method incorrectly uses the lowestSocket member variable, which is initialized to CAM_J and never updated. This results in incorrect extrinsic calculations as it always assumes CAM_J is the lowest socket, despite getLowestSocket() existing to dynamically determine the correct value.

src/device/CalibrationHandler.cpp#L486-L510

depthai-core/src/device/CalibrationHandler.cpp

Lines 486 to 510 in 21a0706

	std::vector<std::vector<float>> extrinsics;
	if(cameraId == lowestSocket) {
	extrinsics = {{1, 0, 0, 0}, {0, 1, 0, 0}, {0, 0, 1, 0}, {0, 0, 0, 1}};
	} else {
	extrinsics = computeExtrinsicMatrix(cameraId, lowestSocket, false);
	}
	dai::Extrinsics extr;
	for(unsigned int i = 0; i < 3; i++) {
	extr.rotationMatrix.push_back(std::vector<float>());
	for(unsigned int j = 0; j < 3; j++) {
	extr.rotationMatrix[i].push_back(extrinsics[i][j]);
	}
	}
	extr.translation.x = extrinsics[0][3];
	extr.translation.y = extrinsics[1][3];
	extr.translation.z = extrinsics[2][3];
	extr.toCameraSocket = lowestSocket;
	if(cameraId != lowestSocket){
	std::vector<std::vector<float>> specExtrinsics = computeExtrinsicMatrix(cameraId, lowestSocket, true);
	extr.specTranslation.x = specExtrinsics[0][3];
	extr.specTranslation.y = specExtrinsics[1][3];
	extr.specTranslation.z = specExtrinsics[2][3];
	}
	return extr;
	}

Fix in Cursor • Fix in Web

---

Bug: Double Frame Retrieval Causes Inconsistent Outputs

The disparityQueue.get() method is called twice: once to store a frame in the disparity variable, and again when printing its extrinsics. Since get() removes items from the queue, this retrieves two different frames, causing the printed disparity extrinsics to be inconsistent with the disparity variable and other synchronized outputs.

examples/python/StereoDepth/stereo_extrinsics.py#L70-L71

depthai-core/examples/python/StereoDepth/stereo_extrinsics.py

Lines 70 to 71 in 21a0706

	print("Disparity:")
	printExtrinsics(disparityQueue.get().getTransformation().getExtrinsics())

Fix in Cursor • Fix in Web

---

Comment bugbot run to trigger another review on this PR
Was this report helpful? Give feedback by reacting with 👍 or 👎