Ruby on rails 如何防止我的sidekiq工作人员超过其数据库连接限制？

现在我的DigitalOcean管理的数据库后端连接限制是22。以下是我的

sidekiq.yml

和

database.yml

配置：

# config/sidekiq.yml

development:
  :concurrency: 18
production:
  :concurrency: 18

每当我开始一项预定任务时，我都有一批或多名工作人员与数据库进行交互。由于我们计划允许此计划任务同时启动多次，因此我们遇到了许多连接数据库池错误。因此，现在我正试图找出优化这一过程的最佳方法，或者找到另一种可能对我们更好的服务

出于测试目的，我创建了一个sidekiq worker，如下所示：

class MySampleWorker
    include Sidekiq::Worker
    sidekiq_options queue: Rails.env.to_sym
    
    def perform
        User.first
    end
end

如果我同时给这个sidekiq工人打20次电话，一切都会顺利进行。但如果我同时调用它50次，那么我会得到大约3-5个失败的工人，他们最终会被重试

我的问题是--我如何缩放这样的东西？在我的情况下，我将不得不给同一个工人打多次电话，随着需求的增长，电话会越来越多，这显然会导致几个工人失败。在某些情况下，这些工作人员可能每人需要5-10分钟——他们基本上是在远程系统上运行健康检查命令，并等待输出以完成工作人员

以这种方式扩展似乎是灾难性的。与其让工人失败，还有什么方法可以让他们排队，在有可用空间时运行，而不是失败？如果我理解这种工作方式，那么

database.yml

不应该将连接限制在18，因此，当postgresql数据库的连接限制为22时，它不应该尝试一次访问超过18个连接的postgresql数据库吗？我想我应该先看到activerecord数据库连接超时错误，然后再看到postgresql数据库连接问题

下面是我在其中一个工人失败时收到的错误：

2020-06-18T15:05:16.536Z pid=1152049 tid=gofhou0qp WARN: PG::ConnectionBad: FATAL:  remaining connection slots are reserved for non-replication superuser connections

2020-06-18T15:05:16.536Z pid=1152049 tid=gofhou0qp WARN: /usr/local/rvm/gems/ruby-2.5.8/gems/pg-1.2.3/lib/pg.rb:58:in `initialize'
/usr/local/rvm/gems/ruby-2.5.8/gems/pg-1.2.3/lib/pg.rb:58:in `new'
/usr/local/rvm/gems/ruby-2.5.8/gems/pg-1.2.3/lib/pg.rb:58:in `connect'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:692:in `connect'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:223:in `initialize'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:48:in `new'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:48:in `postgresql_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:830:in `new_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:874:in `checkout_new_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:853:in `try_to_checkout_new_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:814:in `acquire_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:538:in `checkout'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:382:in `connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:1033:in `retrieve_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_handling.rb:118:in `retrieve_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_handling.rb:90:in `connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/delegation.rb:76:in `connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/query_methods.rb:934:in `build_arel'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/query_methods.rb:900:in `arel'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:560:in `block in exec_queries'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:584:in `skip_query_cache_if_necessary'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:547:in `exec_queries'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:422:in `load'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:200:in `records'
/usr/local/rvm/gems/ruby-2.5.8/gems/bullet-6.1.0/lib/bullet/active_record52.rb:46:in `records'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:195:in `to_ary'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/finder_methods.rb:532:in `find_nth_with_limit'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/finder_methods.rb:517:in `find_nth'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/finder_methods.rb:125:in `first'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/querying.rb:5:in `first'
/var/www/test-dev/app/workers/my_sample_worker.rb:12:in `perform'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:196:in `execute_job'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:163:in `block in process'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:111:in `local'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/rails.rb:43:in `block in call'
/usr/local/rvm/gems/ruby-2.5.8/gems/activesupport-5.2.4/lib/active_support/execution_wrapper.rb:87:in `wrap'
/usr/local/rvm/gems/ruby-2.5.8/gems/activesupport-5.2.4/lib/active_support/reloader.rb:73:in `block in wrap'
/usr/local/rvm/gems/ruby-2.5.8/gems/activesupport-5.2.4/lib/active_support/execution_wrapper.rb:87:in `wrap'
/usr/local/rvm/gems/ruby-2.5.8/gems/activesupport-5.2.4/lib/active_support/reloader.rb:72:in `wrap'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/rails.rb:42:in `call'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:257:in `stats'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:13:in `call'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:78:in `global'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/logger.rb:10:in `with'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:33:in `prepare'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:162:in `process'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:78:in `process_one'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:68:in `run'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:15:in `watchdog'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:24:in `block in safe_thread'

这是我的彪马配置：

threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
threads threads_count, threads_count

port        ENV.fetch("PORT") { 3000 }

environment ENV.fetch("RAILS_ENV") { "development" }

---

随着您的扩展，您应该只获得一个工作队列，供工作人员处理—如果您的并发设置正确的话。不是失败的工作——你的期望是正确的

但是，将数据库池设置为18的期望可能是错误的。当然，这只适用于每个应用服务器（是否有多个？）

但这将取决于您如何配置Puma，发布该配置

---

有关如何在工作人员和多个应用程序服务器之间共享池的详细说明，请参阅。并且，为了获得设置辅助线程和线程大小的最佳实践。

我的数据库池和sidekiq并发设置为18，但数据库最大连接设置为22。如果我的sidekiq workers设置为与数据库建立50个连接，那么它是否应该在activerecord而不是postgresql中出现错误，因为与postgresql的连接应该受到database.yml文件的限制？还是我误解了？你是用Puma还是Unicorn作为服务器？我用Puma作为服务器我遇到了这个问题，记忆告诉我这是因为池计数没有18那么简单。这也取决于puma配置-puma线程和工作线程的数量。你也可以发布你的puma配置吗？我想你只有一个应用服务器在运行？明白了。由于未设置环境变量RAILS\u MAX\u threads，我的puma config设置为5个线程。更新了包含该配置信息的帖子。只有一台PumaWeb服务器正在运行。我从Sidekiq迁移到Shoryuken，所以我不再需要我的配置，但是那些沼泽帮了我的忙。如果你得到一个明确的答案，请在这里张贴？实际上，您可能需要进行一些实验才能获得最佳结果-并非每个工作人员都会始终使用连接，因此，如果您想要最大的工作人员吞吐量，您可以稍微推动边缘。非常感谢Tom！在使用Digital Ocean的托管数据库实例之后，我设法添加了一个连接池（我认为已经配置好了），这似乎有很大帮助。而以前我不能同时运行20个sidekiq作业，现在看来我可以运行数千个而没有问题。如果我还有什么可以贡献的，我将继续试验和更新。谢谢你的帮助！

threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
threads threads_count, threads_count

port        ENV.fetch("PORT") { 3000 }

environment ENV.fetch("RAILS_ENV") { "development" }