Python 异步dns查找,但结果将转到同一文件
我必须做大量的DNS NAPTR查找,每分钟数千次。 我使用dnspython运行Python脚本,读取一个文件并写回另一个文件。请求速率约为300请求/秒。 我尝试在Python中使用异步DNS,但数字是一样的。 我的脚本可能有缺陷。请看下面。这是Python3.4 但如果结果必须返回到一个文件,甚至可以异步查找吗Python 异步dns查找,但结果将转到同一文件,python,performance,lookup,python-asyncio,Python,Performance,Lookup,Python Asyncio,我必须做大量的DNS NAPTR查找,每分钟数千次。 我使用dnspython运行Python脚本,读取一个文件并写回另一个文件。请求速率约为300请求/秒。 我尝试在Python中使用异步DNS,但数字是一样的。 我的脚本可能有缺陷。请看下面。这是Python3.4 但如果结果必须返回到一个文件,甚至可以异步查找吗 import asyncio import aiodns ... loop = asyncio.get_event_loop() resolver = aiodns.DNSRe
import asyncio
import aiodns
...
loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop)
resolver.nameservers = ['x.y.w.z']
...
@asyncio.coroutine
def getsip(number):
try:
strQuery = str(dns.e164.from_e164("+" + number))
answer = yield from resolver.query(strQuery, 'NAPTR')
for rdata in answer:
return rdata.regex
except:
return ""
with open(filename, 'r') as fread, open(filenameOut, 'w') as fwrite:
reader = csv.DictReader(fread, delimiter='|', quoting=csv.QUOTE_NONE)
reader.fieldnames = fieldnamesIn
writer = csv.DictWriter(fwrite, fieldnames = fieldnamesOut, delimiter='|')
for row in reader:
sys.stdout.write("Processing record number: %d \r" % (total) )
sys.stdout.flush()
total+=1
answer = loop.run_until_complete(getsip(row['NUM']))
if answer == "":
missingAnswers+=1
writer.writerow({'NUM': row['NUM'], 'SIP': answer})
print("Records not found: " + str(missingAnswers) + " of total " + str(total) + " records.")
但如果结果必须返回到一个文件,甚至可以异步查找吗
import asyncio
import aiodns
...
loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop)
resolver.nameservers = ['x.y.w.z']
...
@asyncio.coroutine
def getsip(number):
try:
strQuery = str(dns.e164.from_e164("+" + number))
answer = yield from resolver.query(strQuery, 'NAPTR')
for rdata in answer:
return rdata.regex
except:
return ""
with open(filename, 'r') as fread, open(filenameOut, 'w') as fwrite:
reader = csv.DictReader(fread, delimiter='|', quoting=csv.QUOTE_NONE)
reader.fieldnames = fieldnamesIn
writer = csv.DictWriter(fwrite, fieldnames = fieldnamesOut, delimiter='|')
for row in reader:
sys.stdout.write("Processing record number: %d \r" % (total) )
sys.stdout.flush()
total+=1
answer = loop.run_until_complete(getsip(row['NUM']))
if answer == "":
missingAnswers+=1
writer.writerow({'NUM': row['NUM'], 'SIP': answer})
print("Records not found: " + str(missingAnswers) + " of total " + str(total) + " records.")
如果您不关心结果的顺序,那么实现异步查找很简单。例如,您可以使用计划所有协同路由并行运行,并在每个协同路由完成时收到通知:
@asyncio.coroutine
def process():
with open(filename, 'r') as fread:
reader = csv.DictReader(fread, delimiter='|', quoting=csv.QUOTE_NONE)
reader.fieldnames = fieldnamesIn
rows = list(reader)
with open(filenameOut, 'w') as fwrite:
writer = csv.DictWriter(fwrite, fieldnames=fieldnamesOut, delimiter='|')
missingAnswers = 0
loop = asyncio.get_event_loop()
tasks = [loop.create_task(getsip(row['NUM'])) for row in rows]
for done_coro in asyncio.as_completed(tasks):
answer = yield from done_coro
if answer == ""
missingAnswers += 1
writer.writerow({'NUM': row['NUM'], 'SIP': answer})
print("Records not found: %d of total %d records"
% (missingAnswers, len(rows)))
loop = asyncio.get_event_loop()
loop.run_until_complete(process())
loop.run_直到_complete被阻塞,所以一次只处理一个请求。您应该使用类似的东西。谢谢,您让我走上正轨,@user4815162342 answer解决了这个问题。顺便说一句,我建议至少升级到Python 3.6。async def/await语法非常好,asyncio也已经成熟了——例如,get_event_循环在协同路由中做了正确的事情,所以您不必到处传递循环。