Python Django:随机查询速度慢-尽管进行了优化
我构建了一个API,它应该从一个大型查询集中返回10个随机选择的结果 我有以下4种型号:Python Django:随机查询速度慢-尽管进行了优化,python,django,Python,Django,我构建了一个API,它应该从一个大型查询集中返回10个随机选择的结果 我有以下4种型号: class ScrapingOperation(models.Model): completed = models.BooleanField(default=False) (...) indexes = [ models.Index(fields=['completed'], name='completed_idx'), models.Index(f
class ScrapingOperation(models.Model):
completed = models.BooleanField(default=False)
(...)
indexes = [
models.Index(fields=['completed'], name='completed_idx'),
models.Index(fields=['trusted'], name='trusted_idx'),
]
@property
def ads(self):
"""returns all ads linked to the searches of this operation"""
return Ad.objects.filter(searches__in=self.searches.all())
class Search(models.Model):
completed = models.BooleanField(default=False)
scraping_operation = models.ForeignKey(
ScrapingOperation,
on_delete=models.CASCADE,
related_name='searches'
)
(...)
class Ad(models.Model):
searches = models.ManyToManyField('scraper.Search', related_name='ads')
(...)
class Label(models.Model):
value = models.Integerfield()
linked_ad = models.OneToOneField(
Ad, on_delete=models.CASCADE, related_name='labels'
)
数据库中当前有400.000个以上的Ad
对象,但平均ScrapingOperation
有14000个Ad
对象链接到它。我希望API从这些+/-14000中返回10个随机结果,这些结果还没有链接的标签
对象(每个操作最多只存在几百个)
因此,必须从包含14000个对象的查询中返回10个随机结果
早期版本只需返回1个结果,但使用了速度慢得多的sort_by('?')
方法。当我必须放大它以返回随机的10个Ad
对象时,我使用了一种新的方法,部分基于
以下是选择(并返回)10个随机对象的代码:
# Get all ads linked to the last completed operation
last_op_ads = ScrapingOperation.objects.filter(completed=True).last().ads
# Get all ads that don't have an label yet
random_ads = last_op_ads.filter(labels__isnull=True)
# Get list ids of all potential ads
id_list = random_ads.values_list('id', flat=True)
id_list = list(id_list)
# Select a random sample of 10, get objects with PK matches
samples = rd.sample(id_list, min(len(id_list), 10))
selected_samples = random_ads.filter(id__in=samples)
return selected_samples
然而,尽管我进行了优化,但这个查询需要10多秒才能完成,从而创建了一个非常慢的API
这种长延迟是随机查询固有的吗?(如果是的话,其他程序员如何处理这个限制呢?)或者我的代码中是否有我遗漏的错误/低效
编辑:根据响应,我在下面包含了原始sql查询
(注意:这些数据在我的本地环境中运行,它只包含我的生产环境所包含数据的5%)
编辑2:我尝试了另一种方法,通过更深的选择相关的参数
random_ads = ScrapingOperation.objects.prefetch_related(
'searches__ads__labels',
).filter(completed=True).last().ads.exclude(
labels__isnull=True
)
id_list = random_ads.values_list('id', flat=True)
id_list = list(id_list)
samples = rd.sample(id_list, min(
len(id_list), 10))
selected_samples = random_ads.filter(
id__in=samples)
return selected_samples
这将生成以下SQL查询:
{'time': '0.008', 'sql': 'SELECT "scraper_search"."id",
"scraper_search"."item_id", "scraper_search"."date_started",
"scraper_search"."date_completed", "scraper_search"."completed",
"scraper_search"."round", "scraper_search"."scraping_operation_id",
"scraper_search"."trusted" FROM "scraper_search" WHERE
"scraper_search"."scraping_operation_id" IN (6)'}
{'time': '0.113', 'sql': 'SELECT ("database_ad_searches"."search_id")
AS "_prefetch_related_val_search_id", "database_ad"."id",
"database_ad"."item_id", "database_ad"."item_state",
"database_ad"."title", "database_ad"."seller_id",
"database_ad"."url", "database_ad"."price",
"database_ad"."transaction_type", "database_ad"."transaction_method",
"database_ad"."first_seen", "database_ad"."last_seen",
"database_ad"."promoted" FROM "database_ad" INNER JOIN
"database_ad_searches" ON ("database_ad"."id" =
"database_ad_searches"."ad_id") WHERE
"database_ad_searches"."search_id" IN (130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160)'}
{'time': '0.041', 'sql': 'SELECT "classifier_label"."id",
"classifier_label"."set_by_id", "classifier_label"."ad_id",
"classifier_label"."date", "classifier_label"."phone_type",
"classifier_label"."seller_type", "classifier_label"."sale_type" FROM
"classifier_label" WHERE "classifier_label"."ad_id" IN (1, 3, 6, 10, 20, 29, 30, 35, 43, (and MANY more of these numbers) ....'}
{'time': '1.498', 'sql': 'SELECT "database_ad"."id" FROM "database_ad"
INNER JOIN "database_ad_searches" ON ("database_ad"."id" = "database_ad_searches"."ad_id") LEFT OUTER JOIN "classifier_label" ON
("database_ad"."id" = "classifier_label"."ad_id") WHERE
("database_ad_searches"."search_id" IN (SELECT U0."id" FROM
"scraper_search" U0 WHERE U0."scraping_operation_id" = 6) AND NOT
("classifier_label"."id" IS NOT NULL))'}
每个ScrapingOperation
“仅”有+/-14000个链接广告,但生产中的广告总数为400.000个(并且还在增长)。上面的所有代码都在我的本地环境(仅包含5%的数据)上返回有效结果,但在生产中的API上返回502个错误 我会先尝试隔离链接广告,然后通过生成的随机列按顺序从中随机抽取10个。我不确定这在生成的sql之外如何有效。当然,我更愿意在任务上创建一个存储过程,因为这显然是一个以随机样本为结尾的数据挖掘操作。。如果没有,请跟随链接,让我知道有多少正在发生。编辑:我会尽力帮助你。谢谢你的回复!它执行7个SQL查询,其中2个与上述代码相关。为了更好的可读性,我在我的问题中加入了它们。哪个数据库,您要比较的字段上不应该有一些索引吗?数据库是PostgreSQL,我为ScrapingOperation对象上的“completed”字段编制了索引。(我已经更新了代码来显示这一点)。但我的大多数查找都是在ForeignKey对象上,我的印象是Django会自动为这些对象创建索引。我对PostgreSQL不是很熟悉,但我相信有一个EXPLAIN
命令可以告诉您发生了什么。
{'time': '0.008', 'sql': 'SELECT "scraper_search"."id",
"scraper_search"."item_id", "scraper_search"."date_started",
"scraper_search"."date_completed", "scraper_search"."completed",
"scraper_search"."round", "scraper_search"."scraping_operation_id",
"scraper_search"."trusted" FROM "scraper_search" WHERE
"scraper_search"."scraping_operation_id" IN (6)'}
{'time': '0.113', 'sql': 'SELECT ("database_ad_searches"."search_id")
AS "_prefetch_related_val_search_id", "database_ad"."id",
"database_ad"."item_id", "database_ad"."item_state",
"database_ad"."title", "database_ad"."seller_id",
"database_ad"."url", "database_ad"."price",
"database_ad"."transaction_type", "database_ad"."transaction_method",
"database_ad"."first_seen", "database_ad"."last_seen",
"database_ad"."promoted" FROM "database_ad" INNER JOIN
"database_ad_searches" ON ("database_ad"."id" =
"database_ad_searches"."ad_id") WHERE
"database_ad_searches"."search_id" IN (130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160)'}
{'time': '0.041', 'sql': 'SELECT "classifier_label"."id",
"classifier_label"."set_by_id", "classifier_label"."ad_id",
"classifier_label"."date", "classifier_label"."phone_type",
"classifier_label"."seller_type", "classifier_label"."sale_type" FROM
"classifier_label" WHERE "classifier_label"."ad_id" IN (1, 3, 6, 10, 20, 29, 30, 35, 43, (and MANY more of these numbers) ....'}
{'time': '1.498', 'sql': 'SELECT "database_ad"."id" FROM "database_ad"
INNER JOIN "database_ad_searches" ON ("database_ad"."id" = "database_ad_searches"."ad_id") LEFT OUTER JOIN "classifier_label" ON
("database_ad"."id" = "classifier_label"."ad_id") WHERE
("database_ad_searches"."search_id" IN (SELECT U0."id" FROM
"scraper_search" U0 WHERE U0."scraping_operation_id" = 6) AND NOT
("classifier_label"."id" IS NOT NULL))'}