在Android中使用webrtc保存视频文件时遇到问题

我正在开发基于webrtc的视频聊天应用程序，目前视频通话正在运行，但我想使用VideoFileRenderer从远程视频流录制视频，该接口有许多实现，例如： 这是我正在使用的实现。它将视频保存到文件中没有问题，但我只能在使用编解码器后在桌面上播放，因为文件是.y4m而不是.mp4，当我尝试使用VideoView播放它时，它说它无法播放视频，即使我尝试使用android附带的视频播放器播放视频，它也无法播放，我只能使用MXPlayer播放它，VLC或桌面上有编解码器的任何其他应用程序

简化问题：

如何在本机android VideoView上播放video.y4m？

我将进一步简化它，我将假设我不了解记录文件的格式，以下是我用于记录文件的代码：

开始录制时：

remoteVideoFileRenderer = new VideoFileRenderer(
                fileToRecordTo.getAbsolutePath(),
                640,
                480,
                rootEglBase.getEglBaseContext());
        remoteVideoTrack.addSink(remoteVideoFileRenderer);

remoteVideoFileRenderer.release();

完成录制时：

remoteVideoFileRenderer = new VideoFileRenderer(
                fileToRecordTo.getAbsolutePath(),
                640,
                480,
                rootEglBase.getEglBaseContext());
        remoteVideoTrack.addSink(remoteVideoFileRenderer);

remoteVideoFileRenderer.release();

现在问题又来了：我有一个“fileToRecordTo”，这个视频文件可以在GOM（windows）、VLC（windows、mac和Android）、MXPlayer（Android）上播放，但我既不能使用嵌入Android的播放器（如果可以，我会在我的应用程序中使用这个播放器）也不能在Android原生videoView上播放

任何帮助。

仅视频录制

我的项目中也有类似的案例。起初，我尝试了WebRTC的默认VideoFileRenderer，但视频大小太大，因为没有应用压缩。 我找到了这个存储库。这对我来说真的很有帮助。

这是一个循序渐进的指南。我也做了一些调整

将此类添加到项目中。它有很多选项来配置最终的视频格式

import android.media.MediaCodec;
import android.media.MediaCodecInfo;
import android.media.MediaFormat;
import android.media.MediaMuxer;
import android.os.Handler;
import android.os.HandlerThread;
import android.util.Log;
import android.view.Surface;

import org.webrtc.EglBase;
import org.webrtc.GlRectDrawer;
import org.webrtc.VideoFrame;
import org.webrtc.VideoFrameDrawer;
import org.webrtc.VideoSink;

import java.io.IOException;
import java.nio.ByteBuffer;

class FileEncoder implements VideoSink {
    private static final String TAG = "FileRenderer";
    private final HandlerThread renderThread;
    private final Handler renderThreadHandler;
    private int outputFileWidth = -1;
    private int outputFileHeight = -1;
    private ByteBuffer[] encoderOutputBuffers;
    private EglBase eglBase;
    private EglBase.Context sharedContext;
    private VideoFrameDrawer frameDrawer;
    private static final String MIME_TYPE = "video/avc";    // H.264 Advanced Video Coding
    private static final int FRAME_RATE = 30;               // 30fps
    private static final int IFRAME_INTERVAL = 5;           // 5 seconds between I-frames
    private MediaMuxer mediaMuxer;
    private MediaCodec encoder;
    private MediaCodec.BufferInfo bufferInfo;
    private int trackIndex = -1;
    private boolean isRunning = true;
    private GlRectDrawer drawer;
    private Surface surface;

    FileEncoder(String outputFile, final EglBase.Context sharedContext) throws IOException {
        renderThread = new HandlerThread(TAG + "RenderThread");
        renderThread.start();
        renderThreadHandler = new Handler(renderThread.getLooper());
        bufferInfo = new MediaCodec.BufferInfo();
        this.sharedContext = sharedContext;

        mediaMuxer = new MediaMuxer(outputFile,
                MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
    }

    private void initVideoEncoder() {
        MediaFormat format = MediaFormat.createVideoFormat(MIME_TYPE, 1280, 720);

        format.setInteger(MediaFormat.KEY_COLOR_FORMAT,
                MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
        format.setInteger(MediaFormat.KEY_BIT_RATE, 6000000);
        format.setInteger(MediaFormat.KEY_FRAME_RATE, FRAME_RATE);
        format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, IFRAME_INTERVAL);

        try {
            encoder = MediaCodec.createEncoderByType(MIME_TYPE);
            encoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
            renderThreadHandler.post(() -> {
                eglBase = EglBase.create(sharedContext, EglBase.CONFIG_RECORDABLE);
                surface = encoder.createInputSurface();
                eglBase.createSurface(surface);
                eglBase.makeCurrent();
                drawer = new GlRectDrawer();
            });
        } catch (Exception e) {
            Log.wtf(TAG, e);
        }
    }

    @Override
    public void onFrame(VideoFrame frame) {
        frame.retain();
        if (outputFileWidth == -1) {
            outputFileWidth = frame.getRotatedWidth();
            outputFileHeight = frame.getRotatedHeight();
            initVideoEncoder();
        }
        renderThreadHandler.post(() -> renderFrameOnRenderThread(frame));
    }

    private void renderFrameOnRenderThread(VideoFrame frame) {
        if (frameDrawer == null) {
            frameDrawer = new VideoFrameDrawer();
        }
        frameDrawer.drawFrame(frame, drawer, null, 0, 0, outputFileWidth, outputFileHeight);
        frame.release();
        drainEncoder();
        eglBase.swapBuffers();
    }

    /**
     * Release all resources. All already posted frames will be rendered first.
     */
    void release() {
        isRunning = false;
        renderThreadHandler.post(() -> {
            if (encoder != null) {
                encoder.stop();
                encoder.release();
            }
            eglBase.release();
            mediaMuxer.stop();
            mediaMuxer.release();
            renderThread.quit();
        });
    }

    private boolean encoderStarted = false;
    private volatile boolean muxerStarted = false;
    private long videoFrameStart = 0;

    private void drainEncoder() {
        if (!encoderStarted) {
            encoder.start();
            encoderOutputBuffers = encoder.getOutputBuffers();
            encoderStarted = true;
            return;
        }
        while (true) {
            int encoderStatus = encoder.dequeueOutputBuffer(bufferInfo, 10000);
            if (encoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
                break;
            } else if (encoderStatus == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
                // not expected for an encoder
                encoderOutputBuffers = encoder.getOutputBuffers();
                Log.e(TAG, "encoder output buffers changed");
            } else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
                // not expected for an encoder
                MediaFormat newFormat = encoder.getOutputFormat();

                Log.e(TAG, "encoder output format changed: " + newFormat);
                trackIndex = mediaMuxer.addTrack(newFormat);
                if (!muxerStarted) {
                    mediaMuxer.start();
                    muxerStarted = true;
                }
                if (!muxerStarted)
                    break;
            } else if (encoderStatus < 0) {
                Log.e(TAG, "unexpected result fr om encoder.dequeueOutputBuffer: " + encoderStatus);
            } else { // encoderStatus >= 0
                try {
                    ByteBuffer encodedData = encoderOutputBuffers[encoderStatus];
                    if (encodedData == null) {
                        Log.e(TAG, "encoderOutputBuffer " + encoderStatus + " was null");
                        break;
                    }
                    // It's usually necessary to adjust the ByteBuffer values to match BufferInfo.
                    encodedData.position(bufferInfo.offset);
                    encodedData.limit(bufferInfo.offset + bufferInfo.size);
                    if (videoFrameStart == 0 && bufferInfo.presentationTimeUs != 0) {
                        videoFrameStart = bufferInfo.presentationTimeUs;
                    }
                    bufferInfo.presentationTimeUs -= videoFrameStart;
                    if (muxerStarted)
                        mediaMuxer.writeSampleData(trackIndex, encodedData, bufferInfo);
                    isRunning = isRunning && (bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) == 0;
                    encoder.releaseOutputBuffer(encoderStatus, false);
                    if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
                        break;
                    }
                } catch (Exception e) {
                    Log.wtf(TAG, e);
                    break;
                }
            }
        }
    }

    private long presTime = 0L;

}

当您收到要录制的流（远程或本地）时，可以初始化录制

FileEncoder recording = new FileEncoder("path/to/video", rootEglBase.eglBaseContext)
remoteVideoTrack.addSink(recording)

remoteVideoTrack.removeSink(recording)
recording.release()

通话结束后，您需要停止并释放录音

FileEncoder recording = new FileEncoder("path/to/video", rootEglBase.eglBaseContext)
remoteVideoTrack.addSink(recording)

remoteVideoTrack.removeSink(recording)
recording.release()

这足以录制视频，但没有音频

视频和音频录制 要录制本地对等方的音频，您需要使用此类（）。但首先需要设置AudioDeviceModule对象

AudioDeviceModule adm = createJavaAudioDevice()
peerConnectionFactory = PeerConnectionFactory.builder()
    .setOptions(options)
    .setAudioDeviceModule(adm)
    .setVideoEncoderFactory(defaultVideoEncoderFactory)
    .setVideoDecoderFactory(defaultVideoDecoderFactory)
    .createPeerConnectionFactory()
adm.release()

private AudioDeviceModule createJavaAudioDevice() {
    //Implement AudioRecordErrorCallback
    //Implement AudioTrackErrorCallback
 return JavaAudioDeviceModule.builder(this)
    .setSamplesReadyCallback(audioRecorder)
    //Default audio source is Voice Communication which is good for VoIP sessions. You can change to the audio source you want.
    .setAudioSource(MediaRecorder.AudioSource.VOICE_COMMUNICATION)
    .setAudioRecordErrorCallback(audioRecordErrorCallback)
    .setAudioTrackErrorCallback(audioTrackErrorCallback)
    .createAudioDeviceModule()

}

合并音频和视频

添加此依赖项

implementation 'com.googlecode.mp4parser:isoparser:1.1.22'

然后在通话结束时将此片段添加到代码中。确保视频和音频录制已正确停止和释放

try {
     Movie video;
     video = MovieCreator.build("path/to/recorded/video");
     Movie audio;
     audio = MovieCreator.build("path/to/recorded/audio");
     Track audioTrack = audio.getTracks().get(0)  
     video.addTrack(audioTrack);
     Container out = new DefaultMp4Builder().build(video);
     FileChannel fc = new FileOutputStream(new File("path/to/final/output")).getChannel();
     out.writeContainer(fc);
     fc.close();
} catch (IOException e) {
     e.printStackTrace();
}

---

我知道这不是在Android WebRTC视频通话中录制音频和视频的最佳解决方案。如果有人知道如何使用WebRTC提取音频，请添加评论

同样的问题，你是否想过使用“MediaMuxer”转换原始帧，然后将其保存到mp4而不是y4m？我想了想，然后我发现该文件只有视频，但没有音频，我不知道如何获取音频文件，然后我会尝试这样做，如果成功，我会添加答案，如果你知道怎么做，请帮助我@AviramFireberger即使你没有做，但是如果你只通过视频来做，请分享你是如何做到的，然后我可能会帮你不要担心@AviramFireberger，我会，一旦做了，我会在这里标记你，问题是我最近有点忙，但我保证我会继续做这件事，完成后会给你贴上标签，我甚至会尝试通过电子邮件直接与你联系。你在RTC通话中能录制音频吗？你真的做得很好，伙计。@Oliver你从哪里获得remoteVideoTrack？@Oliver我必须添加FileEncoder录制=新的FileEncoder（“路径/到/视频”，rootEglBase.eglBaseContext）remoteVideoTrack.addSink（录音）在CallActivity类中。嘿@Oliver想知道这里收集的音频是来自他人的（通过webrtc），还是来自麦克风输入？