Python django-多线程查询

我使用的是django 1.8，我的逻辑要求在循环中进行查询。 我有多个级别的外键： 卡车->箱子->瓶子

因此，多个级联for循环。 下面的代码过于简化，但使其更快的唯一方法似乎是并行化主循环

我需要加快速度，所以我使用了多处理，如下所示。 查询是只读的！ 这段带有sqlite和postgres的代码陷入死锁

sqlite3上的类似代码失败，原因是：

DatabaseError: database disk image is malformed

我需要知道如何加快速度。 如果我需要将模型对象放入锁中，那么我应该如何做才能避免太多的锁

视图.py

from django.shortcuts import render
from django.contrib.auth import authenticate, login, logout
from django.contrib.auth.decorators import login_required
from django.contrib.admin.views.decorators import staff_member_required
from django.http import HttpResponseRedirect

from models import Truck, Box, Bottle

from multiprocessing import Pool
from functools import partial
MULTI_THREAD = True

def get_data_per_bottle(box):
    data = Bottle.objects.filter(box=box).values_list('color')
    data = [x[0] for x in data]
    return data

def get_data_per_box(some_param, truck):
    rows = []
    for bottle in Box.objects.filter(truck=truck):
        rows.append(get_data_per_bottle(bottle))
    return rows

def get_rows(location):
    trucks = Truck.objects.filter(location=location)
    some_param = ''
    data = []
    if MULTI_THREAD:
        pool = Pool(processes=2)
        func = partial(get_data_per_box, some_param)
        data = pool.map(func, trucks)
        pool.close()
        pool.join()
    else:
        for truck in trucks:
            data.append(get_data_per_box(some_param, truck))

    return data

def index(request):
    rows = get_rows('usa')
    return render(request, 'index.html', locals())

from django.db import models

class Bottle(models.Model):
    color = models.CharField(max_length=30)

class Box(models.Model):
    bottle = models.ForeignKey(Bottle)

class Truck(models.Model):
    box = models.ForeignKey(Box)
    location = models.CharField(max_length=30)

型号.py

from django.shortcuts import render
from django.contrib.auth import authenticate, login, logout
from django.contrib.auth.decorators import login_required
from django.contrib.admin.views.decorators import staff_member_required
from django.http import HttpResponseRedirect

from models import Truck, Box, Bottle

from multiprocessing import Pool
from functools import partial
MULTI_THREAD = True

def get_data_per_bottle(box):
    data = Bottle.objects.filter(box=box).values_list('color')
    data = [x[0] for x in data]
    return data

def get_data_per_box(some_param, truck):
    rows = []
    for bottle in Box.objects.filter(truck=truck):
        rows.append(get_data_per_bottle(bottle))
    return rows

def get_rows(location):
    trucks = Truck.objects.filter(location=location)
    some_param = ''
    data = []
    if MULTI_THREAD:
        pool = Pool(processes=2)
        func = partial(get_data_per_box, some_param)
        data = pool.map(func, trucks)
        pool.close()
        pool.join()
    else:
        for truck in trucks:
            data.append(get_data_per_box(some_param, truck))

    return data

def index(request):
    rows = get_rows('usa')
    return render(request, 'index.html', locals())

from django.db import models

class Bottle(models.Model):
    color = models.CharField(max_length=30)

class Box(models.Model):
    bottle = models.ForeignKey(Bottle)

class Truck(models.Model):
    box = models.ForeignKey(Box)
    location = models.CharField(max_length=30)

---

查看

预回迁相关（）

。我怀疑你是否真的需要多重处理，你只是有太多的查询。@knbk。谢谢它确实加快了一点速度，但仍然有点慢。我不相信您正确使用了

prefetch\u related

。你能展示一下你的实际型号和代码吗？@knbk我已经更新了代码，并把整个示例代码放进去了；当我设置processs=2时，此代码会出现死锁。请查看

prefetch\u related（）

。我怀疑你是否真的需要多重处理，你只是有太多的查询。@knbk。谢谢它确实加快了一点速度，但仍然有点慢。我不相信您正确使用了

prefetch\u related

。你能展示一下你的实际型号和代码吗？@knbk我已经更新了代码，并把整个示例代码放进去了；当我设置processs=2时，这段代码执行死锁