Python 异步dns查找,但结果将转到同一文件

Python 异步dns查找,但结果将转到同一文件,python,performance,lookup,python-asyncio,Python,Performance,Lookup,Python Asyncio,我必须做大量的DNS NAPTR查找,每分钟数千次。 我使用dnspython运行Python脚本,读取一个文件并写回另一个文件。请求速率约为300请求/秒。 我尝试在Python中使用异步DNS,但数字是一样的。 我的脚本可能有缺陷。请看下面。这是Python3.4 但如果结果必须返回到一个文件,甚至可以异步查找吗 import asyncio import aiodns ... loop = asyncio.get_event_loop() resolver = aiodns.DNSRe

我必须做大量的DNS NAPTR查找,每分钟数千次。 我使用dnspython运行Python脚本,读取一个文件并写回另一个文件。请求速率约为300请求/秒。 我尝试在Python中使用异步DNS,但数字是一样的。 我的脚本可能有缺陷。请看下面。这是Python3.4

但如果结果必须返回到一个文件,甚至可以异步查找吗

import asyncio
import aiodns

...

loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop)
resolver.nameservers = ['x.y.w.z']

...

@asyncio.coroutine
def getsip(number):

    try:
        strQuery = str(dns.e164.from_e164("+" + number))
        answer = yield from resolver.query(strQuery, 'NAPTR')

        for rdata in answer:
            return rdata.regex

    except:
        return ""

with open(filename, 'r') as fread, open(filenameOut, 'w') as fwrite:
    reader = csv.DictReader(fread, delimiter='|', quoting=csv.QUOTE_NONE)
    reader.fieldnames = fieldnamesIn

    writer = csv.DictWriter(fwrite, fieldnames = fieldnamesOut, delimiter='|')

    for row in reader:
        sys.stdout.write("Processing record number: %d \r" % (total) )
        sys.stdout.flush()
        total+=1
        answer = loop.run_until_complete(getsip(row['NUM']))
        if answer == "":
            missingAnswers+=1

        writer.writerow({'NUM': row['NUM'], 'SIP': answer})

print("Records not found: " + str(missingAnswers) + " of total " +  str(total) + " records.")
但如果结果必须返回到一个文件,甚至可以异步查找吗

import asyncio
import aiodns

...

loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop)
resolver.nameservers = ['x.y.w.z']

...

@asyncio.coroutine
def getsip(number):

    try:
        strQuery = str(dns.e164.from_e164("+" + number))
        answer = yield from resolver.query(strQuery, 'NAPTR')

        for rdata in answer:
            return rdata.regex

    except:
        return ""

with open(filename, 'r') as fread, open(filenameOut, 'w') as fwrite:
    reader = csv.DictReader(fread, delimiter='|', quoting=csv.QUOTE_NONE)
    reader.fieldnames = fieldnamesIn

    writer = csv.DictWriter(fwrite, fieldnames = fieldnamesOut, delimiter='|')

    for row in reader:
        sys.stdout.write("Processing record number: %d \r" % (total) )
        sys.stdout.flush()
        total+=1
        answer = loop.run_until_complete(getsip(row['NUM']))
        if answer == "":
            missingAnswers+=1

        writer.writerow({'NUM': row['NUM'], 'SIP': answer})

print("Records not found: " + str(missingAnswers) + " of total " +  str(total) + " records.")
如果您不关心结果的顺序,那么实现异步查找很简单。例如,您可以使用计划所有协同路由并行运行,并在每个协同路由完成时收到通知:

@asyncio.coroutine
def process():
    with open(filename, 'r') as fread:
        reader = csv.DictReader(fread, delimiter='|', quoting=csv.QUOTE_NONE)
        reader.fieldnames = fieldnamesIn
        rows = list(reader)

    with open(filenameOut, 'w') as fwrite:
        writer = csv.DictWriter(fwrite, fieldnames=fieldnamesOut, delimiter='|')
        missingAnswers = 0

        loop = asyncio.get_event_loop()
        tasks = [loop.create_task(getsip(row['NUM'])) for row in rows]
        for done_coro in asyncio.as_completed(tasks):
            answer = yield from done_coro
            if answer == ""
                missingAnswers += 1
            writer.writerow({'NUM': row['NUM'], 'SIP': answer})

    print("Records not found: %d of total %d records"
          % (missingAnswers, len(rows)))

loop = asyncio.get_event_loop()
loop.run_until_complete(process())

loop.run_直到_complete被阻塞,所以一次只处理一个请求。您应该使用类似的东西。谢谢,您让我走上正轨,@user4815162342 answer解决了这个问题。顺便说一句,我建议至少升级到Python 3.6。async def/await语法非常好,asyncio也已经成熟了——例如,get_event_循环在协同路由中做了正确的事情,所以您不必到处传递循环。