elasticsearch Scrapy到elasticsearch批量导入?,elasticsearch,scrapy,elasticsearch,Scrapy" /> elasticsearch Scrapy到elasticsearch批量导入?,elasticsearch,scrapy,elasticsearch,Scrapy" />

elasticsearch Scrapy到elasticsearch批量导入?

elasticsearch Scrapy到elasticsearch批量导入?,elasticsearch,scrapy,elasticsearch,Scrapy,我正在尝试将一些数据输出到elasticsearch批量导入格式。这需要两行JL,如下所示: {"index": {"_type": "media", "_id": "https://macaulaylibrary.org/asset/75247", "_index": "audiomnia_dev"}} {"description": "Macaulay Library ML75247; aracari sp.; Pteroglossus sp.; \u00a9\u00a0Curtis Mar

我正在尝试将一些数据输出到elasticsearch批量导入格式。这需要两行JL,如下所示:

{"index": {"_type": "media", "_id": "https://macaulaylibrary.org/asset/75247", "_index": "audiomnia_dev"}}
{"description": "Macaulay Library ML75247; aracari sp.; Pteroglossus sp.; \u00a9\u00a0Curtis Marantz; Lago Sachavacaya Trail, right bank Rio Tambopata, Madre de Dios, Peru; 23 Aug 1994", "creator": "Curtis Marantz", "url": "https://macaulaylibrary.org/asset/75247", "image": "https://macaulaylibrary.org/media/Spectrograms/audio/image/320/0/75/75247.jpg", "commonName": "aracari sp.", "fileFormat": "audio", "sciName": "Pteroglossus sp.", "dateCreated": "1994-08-23T08:13:00", "geo": {"lat": "-12.9", "lon": "-69.3667"}, "contentLocation": "Lago Sachavacaya Trail, right bank Rio Tambopata, Madre de Dios, Peru", "name": "ML75247 aracari sp. Macaulay Library"}
有没有一种方法可以在Scrapy中可靠地做到这一点?我有以下几点,但比赛条件发生了,在某些情况下,它打乱了线的顺序,这导致Elasticsearch批量API呕吐:

yield { "index" : {
    "_index" : "audiomnia_dev",
    "_type" : "media",
    "_id" : json_ld["url"] }
}
yield json_ld

确保jl的两行同时仍遵循生成器/屈服模式的正确方法是什么?

让spider屈服一个包含所有相关数据的对象,并编写一个自定义以正确格式化elasticsearch