Opencv 姿势估计-cv:：使用Scenekit解算PNP-坐标系问题

我一直在通过OpenCV的cv:：solvePNP，使用Apple Vision framework的功能/关键点，进行姿势估计（使用图像上的2D点校正3D模型上的关键点以匹配姿势）

TL-DR:

我的场景工具包模型正在被平移，当从solvePnP内省平移和旋转向量时，单位看起来是正确的（即，它们的数量级正确），但平移的坐标系显示为关闭状态：

我试图通过solvePnP wrt to Metal/OpenGL坐标系和我的相机投影矩阵了解坐标系要求

我的SCNCamera需要什么“projectionMatrix”来匹配传递到solvePnP的基于图像的坐标系

我读到/相信我正在考虑的一些事情

• OpenCV和OpenGL（因此金属）有行主要和列主要的区别
• OpenCV的3D坐标系不同于OpenGL（因此金属）

更长的代码：

我的工作流程如下：

第1步-使用3D模型工具内省3D模型上的点，并获取2D检测特征中主要关键点的对象顶点位置。我用的是左瞳孔，右瞳孔，鼻尖，下巴尖，左外唇角，右外唇角

步骤2-运行vision请求并提取图像空间中的点列表（转换为OpenCV的左上角坐标系），然后提取相同的2D点有序列表

步骤3-使用输入图像的大小构造相机矩阵

步骤4-运行cv:：solvePnP，然后使用cv:：Rodrigues将旋转向量转换为矩阵

步骤5-将生成的变换的坐标系转换为适合GPU的坐标系-反转y轴和z轴，将平移和旋转组合为单个4x4矩阵，然后将其转换为OpenGL/Metal的适当主度

步骤6-通过以下方式将生成的变换应用于Scenekit：

        let faceNodeTransform = openCVWrapper.transform(for: landmarks, imageSize: size)
        self.destinationView.pointOfView?.transform = SCNMatrix4Invert(faceNodeTransform)

下面是我的Obj-C++OpenCV包装器，它接收视觉地标的子集和正在查看的图像的真实像素大小：

/ https://answers.opencv.org/question/23089/opencv-opengl-proper-camera-pose-using-solvepnp/
- (SCNMatrix4) transformFor:(VNFaceLandmarks2D*)landmarks imageSize:(CGSize)imageSize
{
    // 1 convert landmarks to image points in image space (pixels) to vector of cv::Point2f's : 
    // Note that this translates the point coordinate system to be top left oriented for OpenCV's image coordinates:
    std::vector<cv::Point2f >  imagePoints = [self imagePointsForLandmarks:landmarks imageSize:imageSize];

    // 2 Load Model Points
    std::vector<cv::Point3f >  modelPoints = [self modelPoints];

    // 3 create our camera extrinsic matrix
    // TODO - see if this is sane?
    double max_d = fmax(imageSize.width, imageSize.height);
    cv::Mat cameraMatrix = (cv::Mat_<double>(3,3) << max_d, 0, imageSize.width/2.0,
                 0,    max_d, imageSize.height/2.0,
                 0,    0,    1.0);

    // 4 Run solvePnP
    double distanceCoef[] = {0,0,0,0};
    cv::Mat distanceCoefMat = cv::Mat(1 ,4 ,CV_64FC1,distanceCoef);

    // Output Matrixes
    std::vector<double> rv(3);
    cv::Mat rotationOut = cv::Mat(rv);

    std::vector<double> tv(3);
    cv::Mat translationOut = cv::Mat(tv);

    cv::solvePnP(modelPoints, imagePoints, cameraMatrix, distanceCoefMat, rotationOut, translationOut, false, cv::SOLVEPNP_EPNP);

    // 5 Convert rotation matrix (actually a vector)
    // To a real 4x4 rotation matrix:
    cv::Mat viewMatrix = cv::Mat::zeros(4, 4, CV_64FC1);
    cv::Mat rotation;
    cv::Rodrigues(rotationOut, rotation);

    // Append our transforms to our matrix and set final to identity:
    for(unsigned int row=0; row<3; ++row)
    {
        for(unsigned int col=0; col<3; ++col)
        {
            viewMatrix.at<double>(row, col) = rotation.at<double>(row, col);
        }
        viewMatrix.at<double>(row, 3) = translationOut.at<double>(row, 0);
    }

    viewMatrix.at<double>(3, 3) = 1.0f;

    // Transpose OpenCV to OpenGL coords
    cv::Mat cvToGl = cv::Mat::zeros(4, 4, CV_64FC1);
    cvToGl.at<double>(0, 0) = 1.0f;
    cvToGl.at<double>(1, 1) = -1.0f; // Invert the y axis
    cvToGl.at<double>(2, 2) = -1.0f; // invert the z axis
    cvToGl.at<double>(3, 3) = 1.0f;
    viewMatrix = cvToGl * viewMatrix;

    // Finally transpose to get correct SCN / OpenGL Matrix :
    cv::Mat glViewMatrix = cv::Mat::zeros(4, 4, CV_64FC1);
    cv::transpose(viewMatrix , glViewMatrix);

    return [self convertCVMatToMatrix4:glViewMatrix];
}

- (SCNMatrix4) convertCVMatToMatrix4:(cv::Mat)matrix
{
    SCNMatrix4 scnMatrix = SCNMatrix4Identity;

    scnMatrix.m11 = matrix.at<double>(0, 0);
    scnMatrix.m12 = matrix.at<double>(0, 1);
    scnMatrix.m13 = matrix.at<double>(0, 2);
    scnMatrix.m14 = matrix.at<double>(0, 3);

    scnMatrix.m21 = matrix.at<double>(1, 0);
    scnMatrix.m22 = matrix.at<double>(1, 1);
    scnMatrix.m23 = matrix.at<double>(1, 2);
    scnMatrix.m24 = matrix.at<double>(1, 3);

    scnMatrix.m31 = matrix.at<double>(2, 0);
    scnMatrix.m32 = matrix.at<double>(2, 1);
    scnMatrix.m33 = matrix.at<double>(2, 2);
    scnMatrix.m34 = matrix.at<double>(2, 3);

    scnMatrix.m41 = matrix.at<double>(3, 0);
    scnMatrix.m42 = matrix.at<double>(3, 1);
    scnMatrix.m43 = matrix.at<double>(3, 2);
    scnMatrix.m44 = matrix.at<double>(3, 3);

    return (scnMatrix);
}

/https://answers.opencv.org/question/23089/opencv-opengl-proper-camera-pose-using-solvepnp/
-（SCNMatrix4）transformFor:（VNFaceLandmarks2D*）landmarks图像大小：（CGSize）图像大小
{
//1将地标转换为图像空间中的图像点（像素）并转换为cv:：Point2f的向量：
//请注意，这将点坐标系转换为OpenCV图像坐标的左上角方向：
std:：vector imagePoints=[self imagePoints ForlandMarks:landmarks imageSize:imageSize]；
//2个荷载模型点
std:：vector modelPoints=[self modelPoints]；
//3创建我们的相机外部矩阵
//TODO-看看这是否正常？
double max_d=fmax（imageSize.width，imageSize.height）；
cv:：Mat cameraMatrix=（cv:：Mat_u3;（3,3）