Azure service fabric 服务结构有状态服务失败-通过System.Environment.FailFast请求进程终止_Azure Service Fabric

Azure service fabric 服务结构有状态服务失败-通过System.Environment.FailFast请求进程终止

azure-service-fabric

Azure service fabric 服务结构有状态服务失败-通过System.Environment.FailFast请求进程终止,azure-service-fabric,Azure Service Fabric,我们在集群中运行的一个服务结构应用程序没有出现问题，但最近已进入“错误”状态我的同事在[this SO article][1]中提出了一个类似的问题，该问题应该在ServiceFabric版本5.3.311中得到解决但是，我们的服务结构版本现在是5.5.216.0 注意：我们的代码是快速删除和重新创建状态，而不是从中添加/删除键该服务是一个可靠的有状态服务，具有1个分区和3个副本（即一个主分区和两个辅助分区）。该服务实现了一个长时间运行的RunAsync（），并在RunAsync（）期间并

我们在集群中运行的一个服务结构应用程序没有出现问题，但最近已进入“错误”状态

我的同事在[this SO article][1]中提出了一个类似的问题，该问题应该在ServiceFabric版本5.3.311中得到解决

但是，我们的服务结构版本现在是5.5.216.0 注意：我们的代码是快速删除和重新创建状态，而不是从中添加/删除键

该服务是一个可靠的有状态服务，具有1个分区和3个副本（即一个主分区和两个辅助分区）。该服务实现了一个长时间运行的RunAsync（），并在RunAsync（）期间并发地读写许多ReliableDictionary条目

我们注意到主节点上的事件日志中存在以下错误：

Application: RPFabric.Service.Tenant.exe
Framework Version: v4.0.30319
Description: The application requested process termination through System.Environment.FailFast(string message).
Message: Lsn obtained through replication is 634711 and metadata present in deleted state is Name: urn:<StateName>, StateProviderId: 131414644208509332, CreateLSN: 634679, DeleteLSN: 634695 MetadataMode: DelayDelete TransactionId: 0
Stack:
   at System.Environment.FailFast(System.String)
   at Microsoft.ServiceFabric.Replicator.Utility.Assert(Boolean, System.String, ...)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager+<ApplyOnRecoveryAsync>d__89.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.ServiceFabric.Replicator.DynamicStateManager+<ApplyOnRecoveryAsync>d__89, Microsoft.ServiceFabric.Data.Impl, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<ApplyOnRecoveryAsync>d__89 ByRef)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager.ApplyOnRecoveryAsync(Int64, Microsoft.ServiceFabric.Replicator.TransactionBase, Microsoft.ServiceFabric.Replicator.ReplicationMetadata)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager+<OnApplyOnLocalStateAsync>d__f6.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.ServiceFabric.Replicator.DynamicStateManager+<OnApplyOnLocalStateAsync>d__f6, Microsoft.ServiceFabric.Data.Impl, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<OnApplyOnLocalStateAsync>d__f6 ByRef)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager.OnApplyOnLocalStateAsync(Int64, Microsoft.ServiceFabric.Replicator.TransactionBase, System.Fabric.OperationData, System.Fabric.OperationData, Microsoft.ServiceFabric.Replicator.ApplyContext)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager+<OnApplyAsync>d__4c.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.ServiceFabric.Replicator.DynamicStateManager+<OnApplyAsync>d__4c, Microsoft.ServiceFabric.Data.Impl, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<OnApplyAsync>d__4c ByRef)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager.OnApplyAsync(Int64, Microsoft.ServiceFabric.Replicator.TransactionBase, System.Fabric.OperationData, System.Fabric.OperationData, Microsoft.ServiceFabric.Replicator.ApplyContext, Int64)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager+<OnApplyAsync>d__47.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.ServiceFabric.Replicator.DynamicStateManager+<OnApplyAsync>d__47, Microsoft.ServiceFabric.Data.Impl, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<OnApplyAsync>d__47 ByRef)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager.OnApplyAsync(Int64, Microsoft.ServiceFabric.Replicator.TransactionBase, System.Fabric.OperationData, System.Fabric.OperationData, Microsoft.ServiceFabric.Replicator.ApplyContext)
   at Microsoft.ServiceFabric.Replicator.LoggingReplicator+<ApplyCallback>d__ce.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.ServiceFabric.Replicator.LoggingReplicator+<ApplyCallback>d__ce, Microsoft.ServiceFabric.Data.Impl, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<ApplyCallback>d__ce ByRef)
   at Microsoft.ServiceFabric.Replicator.LoggingReplicator.ApplyCallback(Microsoft.ServiceFabric.Replicator.LogRecord)
   at Microsoft.ServiceFabric.Replicator.LoggingReplicator+<Microsoft.ServiceFabric.Replicator.ILogRecordsProcessor.ProcessLoggedRecordAsync>d__ab.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.ServiceFabric.Replicator.LoggingReplicator+<Microsoft.ServiceFabric.Replicator.ILogRecordsProcessor.ProcessLoggedRecordAsync>d__ab, Microsoft.ServiceFabric.Data.Impl, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<Microsoft.ServiceFabric.Replicator.ILogRecordsProcessor.ProcessLoggedRecordAsync>d__ab ByRef)
   at Microsoft.ServiceFabric.Replicator.LoggingReplicator.Microsoft.ServiceFabric.Replicator.ILogRecordsProcessor.ProcessLoggedRecordAsync(Microsoft.ServiceFabric.Replicator.LogRecord)
   at Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+<ProcessTransaction>d__1f.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+<ProcessTransaction>d__1f, Microsoft.ServiceFabric.Data.Impl, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<ProcessTransaction>d__1f ByRef)
   at Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher.ProcessTransaction(System.Collections.Generic.List`1<Microsoft.ServiceFabric.Replicator.TransactionLogRecord>)
   at Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+<SeparateTransactions>d__24.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Start[[Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+<SeparateTransactions>d__24, Microsoft.ServiceFabric.Data.Impl, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<SeparateTransactions>d__24 ByRef)
   at Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher.SeparateTransactions()
   at Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+<ProcessLoggedRecords>d__a.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(Int32)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(Int32)
   at Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+<SeparateTransactions>d__24.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>)
   at Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+<ProcessTransaction>d__1f.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>)
   at Microsoft.ServiceFabric.Replicator.LoggingReplicator+<Microsoft.ServiceFabric.Replicator.ILogRecordsProcessor.ProcessLoggedRecordAsync>d__ab.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
   at Microsoft.ServiceFabric.Replicator.LoggingReplicator+<ApplyCallback>d__ce.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.__Canon)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.__Canon)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager+<OnApplyAsync>d__47.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.__Canon)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.__Canon)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager+<OnApplyAsync>d__4c.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.__Canon)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.__Canon)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager+<OnApplyOnLocalStateAsync>d__f6.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.__Canon)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.__Canon)
   at Microsoft.ServiceFabric.Replicator.DynamicStateManager+<ApplyOnRecoveryAsync>d__89.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>)
   at System.Fabric.Store.TStore`5+<Microsoft.ServiceFabric.Replicator.IStateProvider2.RecoverCheckpointAsync>d__f3[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>)
   at System.Fabric.Store.MetadataManager+<WriteAsync>d__24.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>)
   at System.Fabric.Store.MetadataManager+<WriteAsync>d__2c.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.Finish(Boolean)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

应用程序：RPFabric.Service.Tenant.exe
框架版本：v4.0.30319
描述：应用程序通过System.Environment.FailFast（字符串消息）请求进程终止。
消息：通过复制获得的Lsn为634711，处于已删除状态的元数据为名称：urn:，StateProviderId:131414644208509332，CreateLSN:634679，DeleteLSN:634695 MetadataMode:DelayDelete TransactionId:0
堆栈：
at System.Environment.FailFast（System.String）
位于Microsoft.ServiceFabric.Replicator.Utility.Assert（布尔值，System.String，…）
在Microsoft.ServiceFabric.Replicator.DynamicStateManager+d_u89.MoveNext（）中
在System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.\u Canon，mscorlib，Version=4.0.0.0，Culture=neutral，PublicKeyToken=b77a5c561934e089]]启动[[Microsoft.ServiceFabric.Replicator.DynamicStateManager+d\u 89，Microsoft.ServiceFabric.Data.Impl，Version=5.0.0.0，Culture=neutral，PublicKeyToken=31bf3856ad364e35]]（d_u89 ByRef）
位于Microsoft.ServiceFabric.Replicator.DynamicStateManager.ApplyOnRecoveryAsync（Int64，Microsoft.ServiceFabric.Replicator.TransactionBase，Microsoft.ServiceFabric.Replicator.ReplicationMetadata）
在Microsoft.ServiceFabric.Replicator.DynamicStateManager+d_uF6.MoveNext（）中
在System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.\uu Canon，mscorlib，Version=4.0.0.0，Culture=neutral，PublicKeyToken=b77a5c561934e089]]启动[[Microsoft.ServiceFabric.Replicator.DynamicStateManager+d_uF6，Microsoft.ServiceFabric.Data.Impl，Version=5.0.0.0，Culture=neutral，PublicKeyToken=31bf3856ad364e35]]（d_uF6）
位于Microsoft.ServiceFabric.Replicator.DynamicStateManager.OnApplyOnLocalStateAync（Int64，Microsoft.ServiceFabric.Replicator.TransactionBase，System.Fabric.OperationData，System.Fabric.OperationData，Microsoft.ServiceFabric.Replicator.ApplyContext）
在Microsoft.ServiceFabric.Replicator.DynamicStateManager+d_u4C.MoveNext（）中
在System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.\u Canon，mscorlib，Version=4.0.0.0，Culture=neutral，PublicKeyToken=b77a5c561934e089]]启动[[Microsoft.ServiceFabric.Replicator.DynamicStateManager+d_4c，Microsoft.ServiceFabric.Data.Impl，Version=5.0.0.0，Culture=neutral，PublicKeyToken=31bf3856ad364e35]]（d_u4C）
位于Microsoft.ServiceFabric.Replicator.DynamicStateManager.OnApplySync（Int64、Microsoft.ServiceFabric.Replicator.TransactionBase、System.Fabric.OperationData、System.Fabric.OperationData、Microsoft.ServiceFabric.Replicator.ApplyContext、Int64）
在Microsoft.ServiceFabric.Replicator.DynamicStateManager+d_u47.MoveNext（）中
在System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.\u Canon，mscorlib，Version=4.0.0.0，Culture=neutral，PublicKeyToken=b77a5c561934e089]]。启动[[Microsoft.ServiceFabric.Replicator.DynamicStateManager+d_47，Microsoft.ServiceFabric.Data.Impl，Version=5.0.0.0，Culture=neutral，PublicKeyToken=31bf3856ad364e35]]（d_u47 ByRef）
位于Microsoft.ServiceFabric.Replicator.DynamicStateManager.OnApplySync（Int64，Microsoft.ServiceFabric.Replicator.TransactionBase，System.Fabric.OperationData，System.Fabric.OperationData，Microsoft.ServiceFabric.Replicator.ApplyContext）
在Microsoft.ServiceFabric.Replicator.LoggingReplicator+d_uce.MoveNext（）中
在System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.ServiceFabric.Replicator.LoggingReplicator+d_\ce，Microsoft.ServiceFabric.Data.Impl，版本=5.0.0.0，区域性=中立，PublicKeyToken=31bf3856ad364e35]（d_\ce ByRef）
位于Microsoft.ServiceFabric.Replicator.LoggingReplicator.ApplyCallback（Microsoft.ServiceFabric.Replicator.LogRecord）
在Microsoft.ServiceFabric.Replicator.LoggingReplicator+d_uab.MoveNext（）中
在System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.ServiceFabric.Replicator.LoggingReplicator+d_ab，Microsoft.ServiceFabric.Data.Impl，版本=5.0.0.0，区域性=中立，PublicKeyToken=31bf3856ad364e35]（d_ab ByRef）
在Microsoft.ServiceFabric.Replicator.LoggingReplicator.Microsoft.ServiceFabric.Replicator.ILogRecordsProcessor.ProcessLoggedRecordAsync（Microsoft.ServiceFabric.Replicator.LogRecord）中
在Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+d_u1f.MoveNext（）中
在System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+d_1f，Microsoft.ServiceFabric.Data.Impl，版本=5.0.0.0，区域性=中立，PublicKeyToken=31bf3856ad364e35]（d_1f ByRef）
位于Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher.ProcessTransaction（System.Collections.Generic.List`1）
在Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+d_u24.MoveNext（）中
在System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32，mscorlib，Version=4.0.0.0，Culture=neutral，PublicKeyToken=b77a5c561934e089]]。启动[[Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher+d_24，Microsoft.ServiceFabric.Data.Impl，Version=5.0.0.0，Culture=neutral，PublicKeyToken=31bf3856ad364e35]]（d_24 ByRef）
在Microsoft.ServiceFabric.Replicator.LogRecordsDispatcher.SeparateTransactions（）中
位于Microsoft.ServiceFabric.Replicator.LogRecordsDispat