Python 为什么我的django postgres游标在获取所有行后查询超时?
我正在用django迁移postgres数据。我对一个模式执行原始sql查询,并使用django ORM填充目标模式。原始查询如下所示:Python 为什么我的django postgres游标在获取所有行后查询超时?,python,django,postgresql,psycopg2,Python,Django,Postgresql,Psycopg2,我正在用django迁移postgres数据。我对一个模式执行原始sql查询,并使用django ORM填充目标模式。原始查询如下所示: query = ''' SELECT {columns} FROM tbl_one one INNER JOIN tbl_two two on two.one_id=one.id WHERE one.filter = '{filter}' AND two._created_at >= '{start_dat
query = '''
SELECT {columns}
FROM tbl_one one
INNER JOIN tbl_two two on two.one_id=one.id
WHERE one.filter = '{filter}'
AND two._created_at >= '{start_date} 00:00:00'
ORDER BY two._created_at;
'''.format(columns=', '.join(columns), filter=filter,
start_date=start_date)
if connections['old_schema'].connection is None:
cursor = connections['old_schema'].cursor()
with transaction.atomic(using='old_schema'):
cursor.execute(query)
while True:
rows = cursor.fetchmany(1000)
if not rows:
break
for row in rows:
yield dict(zip(columns, row)))
但是,在非常长的查询中,原始查询会在之后弹出,生成所有行:
Traceback (most recent call last):
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/db/backends/__init__.py", line 139, in _commit
return self.connection.commit()
psycopg2.DatabaseError: SSL SYSCALL error: Connection timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "~/app/management/commands/migrate_cass_data.py", line 166, in handle
for pg in self.get_from_postgres(filter, start_date):
File "~/app/management/commands/migrate_cass_data.py", line 235, in get_from_postgres
yield dict(zip(columns, row)))
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/db/transaction.py", line 316, in __exit__
connection.commit()
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/db/backends/__init__.py", line 171, in commit
self._commit()
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/db/backends/__init__.py", line 139, in _commit
return self.connection.commit()
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/utils/six.py", line 549, in reraise
raise value.with_traceback(tb)
File "~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/db/backends/__init__.py", line 139, in _commit
return self.connection.commit()
django.db.utils.DatabaseError: SSL SYSCALL error: Connection timed out
回溯(最近一次呼叫最后一次):
文件“~/.virtualenvs/venv/local/lib/python3.4/site packages/django/db/backends/\uuuuu init\uuuuuu.py”,第139行,在提交中
返回self.connection.commit()
psycopg2.DatabaseError:SSL系统调用错误:连接超时
上述异常是以下异常的直接原因:
回溯(最近一次呼叫最后一次):
文件“manage.py”,第10行,在
从命令行(sys.argv)执行命令
文件“~/.virtualenvs/venv/local/lib/python3.4/site packages/django/core/management/\uuuuu init\uuuuuu.py”,第399行,从命令行执行
utility.execute()
文件“~/.virtualenvs/venv/local/lib/python3.4/site packages/django/core/management/_init__.py”,第392行,在execute中
self.fetch_命令(子命令)。从_argv(self.argv)运行_
文件“~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/core/management/base.py”,第242行,运行于
self.execute(*args,**选项._dict__;
文件“~/.virtualenvs/venv/local/lib/python3.4/site packages/django/core/management/base.py”,执行中第285行
输出=self.handle(*args,**选项)
文件“~/app/management/commands/migrate\u cass\u data.py”,第166行,在句柄中
对于self中的pg.get_from_postgres(过滤器,开始日期):
文件“~/app/management/commands/migrate_cass_data.py”,第235行,在get_from_postgres中
产量目录(zip(列、行)))
文件“~/.virtualenvs/venv/local/lib/python3.4/site packages/django/db/transaction.py”,第316行,在退出时__
commit()连接
文件“~/.virtualenvs/venv/local/lib/python3.4/site packages/django/db/backends/_init__.py”,第171行,提交
自我承诺
文件“~/.virtualenvs/venv/local/lib/python3.4/site packages/django/db/backends/\uuuuu init\uuuuuu.py”,第139行,在提交中
返回self.connection.commit()
文件“~/.virtualenvs/venv/local/lib/python3.4/site packages/django/db/utils.py”,第99行,在退出时__
6.重新播放(dj_exc_类型、dj_exc_值、回溯)
文件“~/.virtualenvs/venv/local/lib/python3.4/site-packages/django/utils/six.py”,第549行,在reraise中
通过_回溯(tb)提升值
文件“~/.virtualenvs/venv/local/lib/python3.4/site packages/django/db/backends/\uuuuu init\uuuuuu.py”,第139行,在提交中
返回self.connection.commit()
django.db.utils.DatabaseError:SSL系统调用错误:连接超时
同样,当查询结果较小时,它可以正常工作。当结果很大时,查询似乎返回所有结果,但似乎没有完全结束。为什么会发生这种情况?我如何解决它?您是否曾经得到过此问题的答案?我尝试的任何长时间操作都会出现相同的错误。我没有。postgres服务器在GCE中,我想知道GCE网络是否正在关闭连接。你有没有得到这个问题的答案?我尝试的任何长时间操作都会出现相同的错误。我没有。postgres服务器在GCE中,我想知道GCE网络是否正在关闭连接。