C# 如何与原始媒体基础API实时同步视频和音频？ 我试图用原始媒体基础界面： IMFSourceReader < /代码>将音频和视频同步到演示文稿中；代码>EVR合成孔径雷达。我尝试过一些技术，但音频比视频更先进

总结一下视频和音频播放的结构：我有两个缓冲区（音频和视频），每个缓冲区容纳6个样本，每个样本都经过处理，另外一个是从

IMFSourceReader

请求的。每种类型（音频或视频）都包含用于处理缓冲区样本的任务。 根据MSDN的说法，演示时钟很适合使用SAR来改进同步，这就是我在这里要做的

尝试同步：为了尝试在正确的时间显示视频，我正在执行一个延迟，我计算如下：

//Calculates the Media Frame Rate. The Numerator and Denominator were obtained from IMFMediaType from the video file loaded locally by IMFSourceReader.
var FrameRate = Math.Round((Numerator / Denominator)).

//Calculates the delay that will be used to call the next sample. Video processing time is 8 to 11ms.
var Delay = Math.Round((1000 - Video Sample Processing Time) / FrameRate) - 12.

视频样本处理时间=+或-10ms

从每个除法中减去的值（-12）是因为如果我实现结果延迟值，视频将崩溃。在我的测试中，人眼可以感知到的东西，大约13毫秒，是我在与音频同步时获得的最佳结果。它一开始很好，但在音频方面却落后了

我没有太多的代码可以显示。但我将向您展示如何尝试播放从本地文件加载的媒体

视频处理任务： 对于音频，处理过程是相同的，只是调用音频缓冲区

这里的延迟时间是个问题，我不知道应该在什么时候实施。在几次测试中，我注意到超过16ms的延迟开始在音频播放中发出吱吱声。因此，音频延迟I保持为15ms

我曾尝试将音频处理与视频处理同步，但没有得到很好的效果，在播放过程中音频崩溃过多

1更新日期：2019年8月24日 我根据VuVirt提供给我的内容修改了代码，但不幸的是，我在某些方面没有成功。 现在我可以正确地保持音频同步，因为我在正确的渲染时间进行渲染。 不幸的是，视频没有遵循相同的过程。它在播放过程中不断崩溃，正如我所怀疑的，如果使用延迟，它的显示效果可以接受。这让我怀疑，也许EVR没有推迟它应该做的事情

样本的呈现时间如下所示，然后放弃选择样本的过程

视频样本被EVR处理得如此之快，以至于样本缓冲区无法跟踪，在一次测试中，我甚至无法获得EVR渲染缓冲区所用的毫秒时间。速度低于毫秒范围

下面是任务中检查和请求缓冲区样本的代码：

//Variable that will contain the Stream type
StreamType Tp = StreamType.TP_Audio;

//wait for the presentation clock to start.
while (StatusPresentationClock != EnumStatusClock.Started)
{

}

// Create the loop that will be checking and ordering the samples.
while (VideoPlayer.Permission)
{
//Checks if the amount of audio buffer samples is below the total size.
if (BufferSamplesAudio.Count <= (LengthBuffer))
{
                            //Continue
}
else
{
//The buffer has the maximum number of samples. Even taking this delay, at various times evr renders faster.
await Task.Delay(1);
goto Done;
}

//Checks if the audio stream has not reached its end
if(VideoPlayer.EndStreamAudio)
{
//The audio stream is over
SendCon.Send("Audio Task", "Stream End!!");

//Exit
break;
}

//Defines that you are reading an audio sample
AudioRender.ReadingAudioSample = true;

//Calls the method that will request a sample from the reader.
ResultCode Result = RequestNewSample(Tp);

//Check call result
if (Result != ResultCode.SS_OK)
{
//Call Exception
throw new Exception("ERROR");
}
                     
//Waits for the reading process to be completed.
while (AudioRender.ReadingAudioSample)
{
                           
}
                        
Done:;
}

注意：我正在使用DXVA2对IMFSourceReader中的加速进行解码，这可能与播放过程中的视频崩溃有关吗

2更新日期：2019年8月25日 视频崩溃的问题是因为我没有使用IMFSinkWriter渲染视频样本。查看您的信息，我注意到它可以防止样本渲染过快。

---

但视频仍与音频不同步。

音频和视频处理需要单独的线程和队列。每个线程都将等待其队列中第一个样本对应的时间，并在到达该时间时将样本发送给渲染器/写入器。感谢您的回答。我已经有了每个流的线程及其相应的示例队列。通过您的评论方式，您的意思是我必须使用的值（从队列中获得的持续时间和示例演示时间）？如果是这样，我是否应该对样本演示时间实施延迟？我有点迷路了。在呈现样本时，我将如何处理样本？是的，您将样本保留在相应的队列中（样本是引用计数的，如果源使用某些内存池，那么如果队列中的样本太多，则可能会被阻塞）。每个队列必须有一个生产者线程和一个消费者线程。使用者线程轮询队列，并从队列中找到最小的表示时间样本。然后使用者线程从其队列中获取该样本（将其从队列中移除），并将其发送到管道的其余部分（写入器/呈现器）。呈现器将正确延迟发送给它们的每个样本。你不需要拖延。否则，如果您仅使用编写器进行muxing，则需要尽快处理示例。在对代码进行任何更改之前，请告知我是否理解。（生产）线程类似于“工厂”，它们将监控队列中的样本数量，并将为读卡器（IMFSourceReader）请求新的样本。线程（使用者）就像一个“客户机”，它将在两个具有最短呈现时间的缓冲区（音频和视频）中查找，并将其发送到适当的渲染器（SAR或EVR），将其从获取它的缓冲区中排除。让我知道这是不是你想教我的！音频和视频处理需要单独的线程和队列。每个线程都将等待其队列中第一个样本对应的时间，并在到达该时间时将样本发送给渲染器/写入器。感谢您的回答。我已经有了每个流的线程及其相应的示例队列。通过您的评论方式，您的意思是我必须使用的值（从队列中获得的持续时间和示例演示时间）？如果是这样，我是否应该对样本演示时间实施延迟？我有点迷路了。在呈现样本时，我将如何处理样本？是的，您将样本保留在相应的队列中（样本是引用计数的，如果源使用某些内存池，那么如果太多，则可能会被阻塞）

//Variable that will contain the Stream type
StreamType Tp = StreamType.TP_Audio;

//wait for the presentation clock to start.
while (StatusPresentationClock != EnumStatusClock.Started)
{

}

// Create the loop that will be checking and ordering the samples.
while (VideoPlayer.Permission)
{
//Checks if the amount of audio buffer samples is below the total size.
if (BufferSamplesAudio.Count <= (LengthBuffer))
{
                            //Continue
}
else
{
//The buffer has the maximum number of samples. Even taking this delay, at various times evr renders faster.
await Task.Delay(1);
goto Done;
}

//Checks if the audio stream has not reached its end
if(VideoPlayer.EndStreamAudio)
{
//The audio stream is over
SendCon.Send("Audio Task", "Stream End!!");

//Exit
break;
}

//Defines that you are reading an audio sample
AudioRender.ReadingAudioSample = true;

//Calls the method that will request a sample from the reader.
ResultCode Result = RequestNewSample(Tp);

//Check call result
if (Result != ResultCode.SS_OK)
{
//Call Exception
throw new Exception("ERROR");
}
                     
//Waits for the reading process to be completed.
while (AudioRender.ReadingAudioSample)
{
                           
}
                        
Done:;
}

                //Calls the method for creating a Media Sink trigger that will be used to render the video.
                Result = CarenEVR.CreateInstanceVideoRender(SurfaceVideo.Handle, out VideoRender.AtivadorVideoSink);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

                //Calls the method to create an instance of EVR.
                Result = VideoRender.ActivatorVideoSink.ActivateObject(InterfacesMediaFoundation.IID_IMFMediaSink, out ICarenQuery MediaSinkEVR);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

                //Converts the interface to the MediaSink.
                VideoRender.VideoSinkRender = ConvertInterfaceBase<ICaren_MidiaSink>(ref MediaSinkEVR);

                //Get interface that represents the real video renderer
                Result = VideoRender.VideoSinkRender.QueryInterface(InterfacesMediaFoundation.IID_IVideoRenderer, out ICarenQuery OutInterfaceSolictada);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

                //Convert interface to real video renderer
                VideoRender.EVR = ConvertInterfaceBase<ICaren_EVR>(ref OutInterfaceSolictada);

                //Initializes the video renderer in standard mode.
                Result = VideoRender.EVR.InitializeVideoRender(null, null);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

                //Queries the service interface for the video renderer
                Result = VideoRender.EVR.QueryInterface(InterfacesMediaFoundation.IID_IMFGetService, out ICarenQuery OutInterfaceServico);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

                //Converts and defines the interface.
                VideoRender.ServiceInterface = ConvertInterfaceBase<ICaren_ObterInterface>(ref OutInterfaceServico);

                //Calls the method to get the EVR from the service interface exposed by the video renderer.
                Result = VideoRender.ServiceInterface.GetService(MFServiceInterfaces.MR_VIDEO_RENDER_SERVICE, InterfacesMediaFoundation.IID_IMFVideoDisplayControl, out ICarenQuery OutInterfaceDisplayControl);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

                //Converts and define the interface.
                VideoRender.VideoControler = ConvertInterfaceBase<ICaren_ControladorVideo>(ref OutInterfaceDisplayControl);

                //Defines the surface handle that will display the video frames.
                Result = VideoRender.VideoControler.SetHandleSurface(SufarceVideo.Handle);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

                //Sets video window position
                Result = VideoRender.VideoControler.SetVideoPosition(null, new CA_Rectangle() { Rigth = SurfaceVideo.Width, Bottom = SurfaceVideo.Height });

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

                //Get Direct3D Manager
                Result = VideoRender.VideoSinkRender.ConsultarServiceInterface(MFServiceInterfaces.MR_VIDEO_ACCELERATION_SERVICE, InterfacesDirectX.IID_IDirect3DDeviceManager9, out ICarenQuery OutInterfaceD3DManager);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

                //Convert interface
                VideoRender.Direct3DManager = ConvertInterfaceBase<ICaren_Direct3DDeviceManager9>(ref OutInterfaceD3DManager);



                //Sets the presentation clock for the media source.

                //Defines the presentation clock that will be used by the player.
                Result = VideoRender.VideoSinkRender.SetPresentationClock(VideoPlayer.PresentationClock);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }



                //Configures the Media Stream Sink and the sample allocator.

                //Configures the Media Stream Sink and the sample allocator.
                Result = VideoRender.VideoSinkRender.GetStreamSinkByIndex(0, out VideoRender.StreamVideo);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

                //Get media type handler
                Result = VideoRender.StreamVideo.GetMediaTypeHandle(out VideoRender.MediaTypeHandler);

                //Checks for no operation error
                if (Result != ResultCode.SS_OK)
                {
                    //Exit Method
                    goto Done;
                }

 //Create the presentation clock
            ResultCode Result = CarenPresentationClock.CreateIntance(out VideoPlayer.PresentationClock);

            //Check Result of Operation
            if (Result != ResultCode.SS_OK)
            {
                //Get out of the method
                goto Done;
            }

            //Create time source for presentation clock
            Result = CarenPresentationClockSourceTime.CreateIntance(out VideoPlayer.SourcePresentationClock);

            //Check Result of Operation
            if (Result != ResultCode.SS_OK)
            {
                //Get out of the method
                goto Done;
            }

            //Creates the status notification clock for the presentation clock.
            Result = CarenClockStateSink.CreateIntance(out VideoPlayer.NotificationPresentationClock);

            //Check Result of Operation
            if (Result != ResultCode.SS_OK)
            {
                //Get out of the method
                goto Done;
            }


            //Configures the clock events.
            VideoPlayer.NotificationPresentationClock.OnClockStart += NotificationPresentationClock_OnClockStart;
            VideoPlayer.NotificationPresentationClock.OnClockPause += NotificationPresentationClock_OnClockPause;
            VideoPlayer.NotificationPresentationClock.OnClockRestart += NotificationPresentationClock_OnClockRestart;
            VideoPlayer.NotificationPresentationClock.OnClockStop += NotificationPresentationClock_OnClockStop;
            VideoPlayer.NotificationPresentationClock.OnClockSetRate += NotificationPresentationClock_OnClockSetRate;

            //This method will log the events of the generated Clock in the native interface and send to managed code.
            VideoPlayer.NotificationPresentationClock.RegisterCallback();


            //Sets the presentation clock.


            //Sets the clock time source in the presentation clock.
            Result = VideoPlayer.PresentationClock.SetSourceTime(VideoPlayer.SourcePresentationClock);

            //Check Result of Operation
            if (Result != ResultCode.SS_OK)
            {
                //Get out of the method
                goto Done;
            }
       
            //Sets the notification clock.
            Result = VideoPlayer.PresentationClock.RegisterObjectNotification(VideoPlayer.NotificationPresentationClock);

            //Check Result of Operation
            if (Result != ResultCode.SS_OK)
            {
                //Get out of the method
                goto Done;
            }
            
        Done:;
            //Return the result
            return Result;