elasticsearch elasticsearch:cluster\u block\u异常太多\u请求/12/索引只读/允许删除（api）

我正在运行带有多进程索引的spider，我想解决这个问题

我总是收到太多的请求，是否有一种配置我可以更改以允许处理并发请求的阈值更大

这是全部错误

2020-12-29 06:24:13 [scrapy.core.scraper] ERROR: Spider error processing <GET https://github.com/topics/python?o=desc&s=updated&page=5> (referer: None)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/scrapy/spiders/__init__.py", line 90, in _parse
    return self.parse(response, **kwargs)
  File "/usr/src/app/humanoyd/spiders/python_topic.py", line 85, in parse
    print_data(repository)
  File "/usr/src/app/humanoyd/spiders/python_topic.py", line 42, in print_data
    repository.save(repository_doc)
  File "/usr/src/app/humanoyd/model/repositories.py", line 18, in save
    repo.save()
  File "/usr/local/lib/python3.8/site-packages/toute/document.py", line 90, in save
    saved_document = self.get_es(es).index(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 398, in index
    return self.transport.perform_request(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/transport.py", line 392, in perform_request
    raise e
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/transport.py", line 358, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 269, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 315, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.TransportError: TransportError(429, 'cluster_block_exception', 'index [xxxxxx] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];')

df命令的输出

Filesystem    512-blocks      Used Available Capacity iused      ifree %iused  Mounted on
/dev/disk1s6   489825072  22025208  12057544    65%  488254 2448637106    0%   /
devfs                385       385         0   100%     666          0  100%   /dev
/dev/disk1s1   489825072 413862176  12057544    98% 4477496 2444647864    0%   /System/Volumes/Data
/dev/disk1s4   489825072  17879464  12057544    60%       8 2449125352    0%   /private/var/vm
/dev/disk1s5   489825072  21349800  12057544    64%  487049 2448638311    0%   /Volumes/Ohne Titel
map auto_home          0         0         0   100%       0          0  100%   /System/Volumes/Data/home

---

根据您收到的错误消息，您的磁盘已满

flood stage disk watermark [5gb] exceeded on [Z8G4W5UlQYy1HOk0JWGP7w][n1_it][/usr/share/elasticsearch/data/nodes/0] free: 3.8gb[1.6%], all indices on this node will be marked read-only

---

您需要增加该节点的可用磁盘空间。

cat/nodes？v和cat/Index？v的输出是什么？您还可以检查您的ES日志，看看是否收到了高磁盘水印的警报？elasticsearch|1{“类型”：“服务器”，“时间戳”：“2020-12-29T06:40:51701Z”，“级别”：“警告”，“组件”：“o.e.c.r.a.DiskThresholdMonitor”，“群集名”：“对话群集”，“节点名”：“n1_it”，“消息”：“洪泛期磁盘水印[5gb]在[Z8G4W5UlQYy1HOk0JWGP7w][n1_-it][usr/share/elasticsearch/data/nodes/0]上超过免费：3.8gb[1.6%]，此节点上的所有索引都将标记为只读，“cluster.uuid”：“UO5VK64TQOYFIEGIUHLSG”，“node.id”：“Z8G4W5UlQYy1HOk0JWGP7w”}这条消息有点让人困惑。您是否也可以运行

df

命令并用您得到的回答更新您的问题？完成！刚刚更新了QIt，您的两个磁盘似乎已满。请看下面我的答案。我通过禁用磁盘水印分配解决了这个问题。好吧，你只是在推迟这个问题。您可能不会再收到错误，但当磁盘已满时，您将无法再索引任何内容，并且您的集群可能会崩溃。限制是有原因的；-）

flood stage disk watermark [5gb] exceeded on [Z8G4W5UlQYy1HOk0JWGP7w][n1_it][/usr/share/elasticsearch/data/nodes/0] free: 3.8gb[1.6%], all indices on this node will be marked read-only