C# K8S上Redis队列的.Net核心Web API速度缓慢

我开发了一个.net核心web API，在该API中，所有请求首先插入到数据库中，然后生成的请求Id插入到redis队列中

通过负载测试，应用程序可以完美地在任何负载数的VM服务器上托管IIS。当应用程序部署在具有3个POD（每个POD配置1个CPU和4 GB RAM）的K8S上时，当请求大于750时，应用程序性能会下降。加载1000个请求时，有时所有请求都成功执行，但完成所有请求需要2到3分钟，有时很少有请求失败。 当负载大于1000到2000时，失败请求的数量会增加。下面是我随机得到的错误

错误1

No connection is available to service this operation: LPUSH GENEVAREQ; IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=12,Free=32755,Min=4,Max=32767), Local-CPU: n/a \n   at StackExchange.Redis.ConnectionMultiplexer.

错误2

at System.Lazy`1.CreateValue()
at System.Lazy`1.get_Value()
at XYZ.Demo.RedisService.RedisConnectionFactory.Connection() in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\XYZ.Demo.RedisService\RedisConnectionFactory.cs:line 26
at XYZ.Demo.RedisService.RedisQueueService`1..ctor(IRedisConnectionFactory connectionFactory) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\XYZ.Demo.RedisService\RedisQueueService.cs:line 22
at XYZ.DemoAPI.Services.RequestService.EnqueueRequest(String requestId, String name, String queueName, Boolean priority) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\DemoAPI\Services\RequestService.cs:line 205
at XYZ.DemoAPI.Services.RequestService.PostReportRequest(RequestParams p) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\DemoAPI\Services\RequestService.cs:line 127
at XYZ.DemoAPI.Controllers.BaseController`1.PushReportRequest[TResult,TRow,P](P param, IProcessRequest`3 service) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\DemoAPI\Controllers\BaseController.cs:line 54
at XYZ.DemoAPI.Controllers.DemoApiController.GetReport(String businessLine, String category, String firmId, String authorization, String authUserId, Object requestPayload, String requestTrackingId) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\DemoAPI\Controllers\DemoApiController.cs:line 57
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()

错误4

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server) \n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\n   at Microsoft.Data.SqlClient.SqlConnection.Open()\n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)\n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)\n   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.InitializeReader(DbContext _, Boolean result)\n   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()\n   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source)\n   at XYZ.Demo.BuisnessService.ValidateService.GetClusterServer(RequestParams requestParams, String& queueName, String& filePath) in D:\\TeamCity\\buildAgent\\work\\abdf9bd4783a2774\\XYZ.Demo.BuisnessService\\ValidateService.cs:line 220\n   at XYZ.DemoAPI.Services.RequestService.PostReportRequest(RequestParams p) in

错误5

The timeout period elapsed during the post-login phase.  The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections.  The duration spent while attempting to connect to this server was - [Pre-Login] initialization=7; handshake=1069; [Login] initialization=0; authentication=0; [Post-Login] complete=13032;  \n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\n   at Microsoft.Data.SqlClient.SqlConnection.Open()\n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)\n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)\n   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.InitializeReader(DbContext _, Boolean result)\n

错误6

504网关超时
504网关超时
nginx/1.17.10
504网关超时
服务器没有及时响应。

请建议如何解决此问题

The timeout period elapsed during the post-login phase.  The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections.  The duration spent while attempting to connect to this server was - [Pre-Login] initialization=7; handshake=1069; [Login] initialization=0; authentication=0; [Post-Login] complete=13032;  \n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\n   at Microsoft.Data.SqlClient.SqlConnection.Open()\n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)\n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)\n   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.InitializeReader(DbContext _, Boolean result)\n