Java 如何在Android应用程序中正确实现MediaPipe端数据包？您将如何使用Iris Mediapipe解决方案推断虹膜方向？

我希望有人能给我一些想法，或者给我一些关于使用Iris.aar与Mediapipe合作创建定制Android应用程序的进一步阅读材料。我已经翻阅了Mediapipe的官方文档，但发现它有点有限，现在我正在努力取得进展。我一直在尝试为Iris模型添加预期的边数据包，并尝试实时提取特定的地标坐标

我的目标是创建一个开源的凝视方向驱动的文本到语音键盘，用于辅助功能，它使用改进的MediaPipe虹膜解决方案来推断用户的注视方向，以控制应用程序，我真的非常感谢对此提供的任何帮助

以下是我目前的发展计划和进展：

从命令行设置Mediapipe并生成示例DONE

生成用于人脸检测和虹膜跟踪的.aar完成

设置Android Studio以构建Mediapipe应用程序完成

使用.aarDONE构建并测试人脸检测示例应用程序

修改人脸检测示例以使用Iris.aar正在进行中

输出虹膜和眼睛边缘之间的坐标以及虹膜和眼睛之间的距离，以实时估计方向。或者如果可能的话，修改图表和计算器来推断这一点，并重建.aar

将凝视方向集成到应用程序中的控制方案中

一旦实现初始控制，扩展应用程序功能

到目前为止，我已经使用以下构建文件生成了一个Iris.aar， 我构建的.aar是否包含子图和主图的计算器，或者我是否需要向我的aar构建文件中添加其他内容

.aar生成文件：

load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")
mediapipe_aar(
name = "mp_iris_tracking_aar",
calculators = ["//mediapipe/graphs/iris_tracking :iris_tracking_gpu_deps"],
)

目前，我有一个android studio项目，其中包含以下资产和前面提到的Iris.aar

Android Studio Assets:
iris_tracking_gpu.binarypb
face_landmark.tflite
iris_landmark.tflite
face_detection_front.tflite

现在，我只是尝试按原样构建它，以便更好地理解过程，并验证构建环境是否正确设置。我已经成功构建并测试了文档中列出的人脸检测示例，这些示例运行正常，但是在修改项目以利用iris时。aar它构建正确，但运行时崩溃，例外情况是：需要侧数据包“焦距像素”，但未提供

我已经尝试根据媒体管道代表中的Iris示例将焦距代码添加到onCreate中，但我不知道如何修改此代码以使用Iris。aar，是否还有其他文档可以为我指明正确的方向

我需要将这个片段（我认为）集成到人脸检测示例的修改代码中，但不确定如何集成。谢谢你的帮助：）

override-on-resume（）{
super.onResume（）
converter=ExternalTextureConverter（eglManager？.context，NUM_缓冲区）
if（PermissionHelper.cameraPermissionsGranted（此））{
变量旋转：Int=0
if（Build.VERSION.SDK\u INT>=Build.VERSION\u code.R）{
旋转=这个。显示！！。旋转
}否则{
旋转=this.windowManager.defaultDisplay.rotation
}
转换器！！.setRotation（旋转）
转换器！！.setFlipY（垂直翻转帧）
startCamera（旋转）
如果（！haveAddedSidePackets）{
val packetCreator=mediapipeFrameProcessor！！.getPacketCreator（）；
val inputSidePackets=mutableMapOf（）
focalLength=cameraHelper？.focalLength像素！！
Log.i（TAG_MAIN，“启动聚焦长度：${cameraHelper？.focalLength像素！！}”）
inputSidePackets.put(
焦距流名称，
packetCreator.createFloat32（focalLength.width.toFloat（））
)
mediapipeFrameProcessor！！.setInputSidePackets（inputSidePackets）
haveAddedSidePackets=true
val imageSize=cameraHelper！！.imageSize
val校准矩阵=矩阵（）
校准矩阵.setValues(
浮动阵列(
焦距长度。宽度*1.0f，
0.0f，
imageSize.width/2.0f，
0.0f，
焦距高度*1.0f，
imageSize.height/2.0f，
0.0f，
0.0f，
1.0f
)
)
val isInvert=CalibleMatrix.invert（matrixPixels2World）
如果（！isInvert）{
matrixPixels2World=矩阵（）
}
}
转换器！！.setConsumer（mediapipeFrameProcessor）
}
}`

    float focalLength = cameraHelper.getFocalLengthPixels();
    if (focalLength != Float.MIN_VALUE) {
    Packet focalLengthSidePacket = processor.getPacketCreator().createFloat32(focalLength);
    Map<String, Packet> inputSidePackets = new HashMap<>();
    inputSidePackets.put(FOCAL_LENGTH_STREAM_NAME, focalLengthSidePacket);
    processor.setInputSidePackets(inputSidePackets);
    }
    haveAddedSidePackets = true;

Modified Face Tracking AAR example:
package com.example.iristracking;

// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

import android.graphics.SurfaceTexture;
import android.os.Bundle;
import android.util.Log;
import java.util.HashMap;
import java.util.Map;
import androidx.appcompat.app.AppCompatActivity;
import android.util.Size;
import android.view.SurfaceHolder;
import android.view.SurfaceView;
import android.view.View;
import android.view.ViewGroup;
import com.google.mediapipe.components.CameraHelper;
import com.google.mediapipe.components.CameraXPreviewHelper;
import com.google.mediapipe.components.ExternalTextureConverter;
import com.google.mediapipe.components.FrameProcessor;
import com.google.mediapipe.components.PermissionHelper;
import com.google.mediapipe.framework.AndroidAssetUtil;
import com.google.mediapipe.framework.Packet;
import com.google.mediapipe.glutil.EglManager;

/** Main activity of MediaPipe example apps. */
public class MainActivity extends AppCompatActivity {
private static final String TAG = "MainActivity";
private boolean haveAddedSidePackets = false;

private static final String FOCAL_LENGTH_STREAM_NAME = "focal_length_pixel";
private static final String OUTPUT_LANDMARKS_STREAM_NAME = "face_landmarks_with_iris";

private static final String BINARY_GRAPH_NAME = "iris_tracking_gpu.binarypb";
private static final String INPUT_VIDEO_STREAM_NAME = "input_video";
private static final String OUTPUT_VIDEO_STREAM_NAME = "output_video";
private static final CameraHelper.CameraFacing CAMERA_FACING = CameraHelper.CameraFacing.FRONT;

// Flips the camera-preview frames vertically before sending them into FrameProcessor to be
// processed in a MediaPipe graph, and flips the processed frames back when they are displayed.
// This is needed because OpenGL represents images assuming the image origin is at the bottom-left
// corner, whereas MediaPipe in general assumes the image origin is at top-left.
private static final boolean FLIP_FRAMES_VERTICALLY = true;

static {
    // Load all native libraries needed by the app.
    System.loadLibrary("mediapipe_jni");
    System.loadLibrary("opencv_java3");
}

// {@link SurfaceTexture} where the camera-preview frames can be accessed.
private SurfaceTexture previewFrameTexture;
// {@link SurfaceView} that displays the camera-preview frames processed by a MediaPipe graph.
private SurfaceView previewDisplayView;

// Creates and manages an {@link EGLContext}.
private EglManager eglManager;
// Sends camera-preview frames into a MediaPipe graph for processing, and displays the processed
// frames onto a {@link Surface}.
private FrameProcessor processor;
// Converts the GL_TEXTURE_EXTERNAL_OES texture from Android camera into a regular texture to be
// consumed by {@link FrameProcessor} and the underlying MediaPipe graph.
private ExternalTextureConverter converter;

// Handles camera access via the {@link CameraX} Jetpack support library.
private CameraXPreviewHelper cameraHelper;


@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    previewDisplayView = new SurfaceView(this);
    setupPreviewDisplayView();

    // Initialize asset manager so that MediaPipe native libraries can access the app assets, e.g.,
    // binary graphs.
    AndroidAssetUtil.initializeNativeAssetManager(this);

    eglManager = new EglManager(null);
    processor =
            new FrameProcessor(
                    this,
                    eglManager.getNativeContext(),
                    BINARY_GRAPH_NAME,
                    INPUT_VIDEO_STREAM_NAME,
                    OUTPUT_VIDEO_STREAM_NAME);
    processor.getVideoSurfaceOutput().setFlipY(FLIP_FRAMES_VERTICALLY);

    PermissionHelper.checkAndRequestCameraPermissions(this);


}

@Override
protected void onResume() {
    super.onResume();
    converter = new ExternalTextureConverter(eglManager.getContext());
    converter.setFlipY(FLIP_FRAMES_VERTICALLY);
    converter.setConsumer(processor);
    if (PermissionHelper.cameraPermissionsGranted(this)) {
        startCamera();
    }
}

@Override
protected void onPause() {
    super.onPause();
    converter.close();
}

@Override
public void onRequestPermissionsResult(
        int requestCode, String[] permissions, int[] grantResults) {
    super.onRequestPermissionsResult(requestCode, permissions, grantResults);
    PermissionHelper.onRequestPermissionsResult(requestCode, permissions, grantResults);
}

private void setupPreviewDisplayView() {
    previewDisplayView.setVisibility(View.GONE);
    ViewGroup viewGroup = findViewById(R.id.preview_display_layout);
    viewGroup.addView(previewDisplayView);

    previewDisplayView
            .getHolder()
            .addCallback(
                    new SurfaceHolder.Callback() {
                        @Override
                        public void surfaceCreated(SurfaceHolder holder) {
                            processor.getVideoSurfaceOutput().setSurface(holder.getSurface());
                        }

                        @Override
                        public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
                            // (Re-)Compute the ideal size of the camera-preview display (the area that the
                            // camera-preview frames get rendered onto, potentially with scaling and rotation)
                            // based on the size of the SurfaceView that contains the display.
                            Size viewSize = new Size(width, height);
                            Size displaySize = cameraHelper.computeDisplaySizeFromViewSize(viewSize);

                            // Connect the converter to the camera-preview frames as its input (via
                            // previewFrameTexture), and configure the output width and height as the computed
                            // display size.
                            converter.setSurfaceTextureAndAttachToGLContext(
                                    previewFrameTexture, displaySize.getWidth(), displaySize.getHeight());
                        }

                        @Override
                        public void surfaceDestroyed(SurfaceHolder holder) {
                            processor.getVideoSurfaceOutput().setSurface(null);
                        }
                    });
}

private void startCamera() {
    cameraHelper = new CameraXPreviewHelper();
    cameraHelper.setOnCameraStartedListener(
            surfaceTexture -> {
                previewFrameTexture = surfaceTexture;
                // Make the display view visible to start showing the preview. This triggers the
                // SurfaceHolder.Callback added to (the holder of) previewDisplayView.
                previewDisplayView.setVisibility(View.VISIBLE);
            });
    cameraHelper.startCamera(this, CAMERA_FACING, /*surfaceTexture=*/ null);

}
}

override fun onResume() {
        super.onResume()
        converter = ExternalTextureConverter(eglManager?.context, NUM_BUFFERS)

        if (PermissionHelper.cameraPermissionsGranted(this)) {
            var rotation: Int = 0
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
                rotation = this.display!!.rotation
            } else {
                rotation = this.windowManager.defaultDisplay.rotation
            }

            converter!!.setRotation(rotation)
            converter!!.setFlipY(FLIP_FRAMES_VERTICALLY)

            startCamera(rotation)

            if (!haveAddedSidePackets) {
                val packetCreator = mediapipeFrameProcessor!!.getPacketCreator();
                val inputSidePackets = mutableMapOf<String, Packet>()

                focalLength = cameraHelper?.focalLengthPixels!!
                Log.i(TAG_MAIN, "OnStarted focalLength: ${cameraHelper?.focalLengthPixels!!}")
                inputSidePackets.put(
                    FOCAL_LENGTH_STREAM_NAME,
                    packetCreator.createFloat32(focalLength.width.toFloat())
                )
                mediapipeFrameProcessor!!.setInputSidePackets(inputSidePackets)
                haveAddedSidePackets = true

                val imageSize = cameraHelper!!.imageSize
                val calibrateMatrix = Matrix()
                calibrateMatrix.setValues(
                    floatArrayOf(
                        focalLength.width * 1.0f,
                        0.0f,
                        imageSize.width / 2.0f,
                        0.0f,
                        focalLength.height * 1.0f,
                        imageSize.height / 2.0f,
                        0.0f,
                        0.0f,
                        1.0f
                    )
                )
                val isInvert = calibrateMatrix.invert(matrixPixels2World)
                if (!isInvert) {
                    matrixPixels2World = Matrix()
                }
            }
            converter!!.setConsumer(mediapipeFrameProcessor)
        }
    }`