Python 在twisted中使用reactor不断扫描数据库表时出现以下错误

在成功运行几个小时后，我遇到以下错误

Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.6/dist-packages/twisted/python/threadpool.py", line 210, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/lib/python2.6/dist-packages/twisted/enterprise/adbapi.py", line 436, in _runInteraction conn.rollback() File "/usr/lib/python2.6/dist-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() _mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')

---

有人能告诉我为什么会出现此错误吗？

有人能告诉我为什么会出现此错误。

因为您做错了

runInteraction

使用游标参数运行提供的函数，该参数指向在线程中运行的事务。您不应该调用reactor.callInThread（Scanner（）.deferExecute）

• 最好使用twisted.internet.task.LoopingCall，它将确保在触发下一个调用之前完成调用
• 在示例中，您只是在运行一个查询，因此可以使用ConnectionPool.runQuery而不是ConnectionPool.runInteraction
• 使用errorBack函数报告异常

试图更正格式不正确的代码，我认为您得到了以下结果：

from twisted.internet import reactor, defer,threads
from twisted.enterprise import adbapi

dbpool = adbapi.ConnectionPool("MySQLdb", '192.168.1.102','test', 'test', 'test')

class Scanner:
  def _execQuery(self,txn):
    sql="SELECT tool_id,tool_name FROM tool_master"
    txn.execute(sql)
    result = txn.fetchall()
    return result

  def objCursor(self):
    return dbpool.runInteraction(self._execQuery)

  def printResult(self,result):
    print "resssssssssssssssssss",result
    reactor.callLater(3,self.deferExecute)

  def deferExecute(self):
    self.objCursor().addCallback(self.printResult)

Scanner()

class MyApp(object):
  reactor.callInThread(Scanner().deferExecute)
  reactor.run()

MyApp()

当您可能需要以下内容时。如果您计划编写twisted

应用程序

将很容易修改此

扫描仪

类以从

twisted.Application.service.service

继承

from twisted.internet import reactor, defer, task
from twisted.enterprise import adbapi

class Scanner(object):
  def __init__(self,dbpool=None):
    self.dbpool = dbpool
    self.loopCall = task.LoopingCall(self.myQuery)
  def start(self):
    print "Started scanner"
    self.loopCall.start(3)
  def stop(self):
    print "Stopping scanner"
    self.loopCall.stop()
  def myQuery(self):
    def interact(txn):
      sql="SELECT tool_id,tool_name FROM tool_master"
      txn.execute(sql)
      return txn.fetchall()
    d = self.dbpool.runInteraction(interact)
    d.addCallbacks(self.printResult,self.printError)
  def printResult(self,result):
    print "Got Result: %r" % result
  def printError(self,error):
    print "Got Error: %r" % error
    error.printTraceback()

if __name__ == '__main__':
  from twisted.internet import reactor
  dbpool = adbapi.ConnectionPool("MySQLdb", '192.168.1.102','test', 'test', 'test')
  s = Scanner(dbpool)
  reactor.callWhenRunning(s.start)
  reactor.addSystemEventTrigger('before','shutdown',s.stop)
  reactor.run()

---

有人能告诉我为什么会出现这个错误吗？

因为你做错了

runInteraction

使用游标参数运行提供的函数，该参数指向在线程中运行的事务。您不应该调用reactor.callInThread（Scanner（）.deferExecute）

• 最好使用twisted.internet.task.LoopingCall，它将确保在触发下一个调用之前完成调用
• 在示例中，您只是在运行一个查询，因此可以使用ConnectionPool.runQuery而不是ConnectionPool.runInteraction
• 使用errorBack函数报告异常

试图更正格式不正确的代码，我认为您得到了以下结果：

from twisted.internet import reactor, defer,threads
from twisted.enterprise import adbapi

dbpool = adbapi.ConnectionPool("MySQLdb", '192.168.1.102','test', 'test', 'test')

class Scanner:
  def _execQuery(self,txn):
    sql="SELECT tool_id,tool_name FROM tool_master"
    txn.execute(sql)
    result = txn.fetchall()
    return result

  def objCursor(self):
    return dbpool.runInteraction(self._execQuery)

  def printResult(self,result):
    print "resssssssssssssssssss",result
    reactor.callLater(3,self.deferExecute)

  def deferExecute(self):
    self.objCursor().addCallback(self.printResult)

Scanner()

class MyApp(object):
  reactor.callInThread(Scanner().deferExecute)
  reactor.run()

MyApp()

当您可能需要以下内容时。如果您计划编写twisted

应用程序

将很容易修改此

扫描仪

类以从

twisted.Application.service.service

继承

from twisted.internet import reactor, defer, task
from twisted.enterprise import adbapi

class Scanner(object):
  def __init__(self,dbpool=None):
    self.dbpool = dbpool
    self.loopCall = task.LoopingCall(self.myQuery)
  def start(self):
    print "Started scanner"
    self.loopCall.start(3)
  def stop(self):
    print "Stopping scanner"
    self.loopCall.stop()
  def myQuery(self):
    def interact(txn):
      sql="SELECT tool_id,tool_name FROM tool_master"
      txn.execute(sql)
      return txn.fetchall()
    d = self.dbpool.runInteraction(interact)
    d.addCallbacks(self.printResult,self.printError)
  def printResult(self,result):
    print "Got Result: %r" % result
  def printError(self,error):
    print "Got Error: %r" % error
    error.printTraceback()

if __name__ == '__main__':
  from twisted.internet import reactor
  dbpool = adbapi.ConnectionPool("MySQLdb", '192.168.1.102','test', 'test', 'test')
  s = Scanner(dbpool)
  reactor.callWhenRunning(s.start)
  reactor.addSystemEventTrigger('before','shutdown',s.stop)
  reactor.run()

---

在Matt的所有建议和帮助之后，我有以下代码正在成功运行：

#!usr/bin/python
# Using the "dbmodule" from the previous example, create a ConnectionPool
from twisted.internet import reactor
from twisted.enterprise import adbapi
from twisted.internet import reactor, defer,threads
from twisted.python.threadpool import ThreadPool
import itertools
from twisted.internet.threads import deferToThread
from twisted.internet import reactor, defer, task
from tools.printTime import *
from tools.getVersion import *
from sh_log import *


concurrent = 30
finished=itertools.count(1)
reactor.suggestThreadPoolSize(concurrent)


#Creating Global Instance variables
path="tools"
lo=Log()

class ToolsBuilder:

  def build(self,txn,tool,asset_id):
    if tool:
        print "\n"
        try:
            sql="select tool_filename from tool_master where tool_id = %s" %(tool,)
            sql_asset="select asset_url from asset_master where asset_id = %s" %(asset_id,)
            txn.execute(sql_asset)
            asset_url = txn.fetchall()
            log_date=lo.log_date()
            txn.execute(sql)
            result = txn.fetchall()

            log='\n'+log_date+"::"+str(result[0][0])+ " tool object is created......\n"
            lo.wfile(log)
            temp=(path +'/' + str(result[0][0]))

            if result:
                if temp:
                    f=open(temp).read()
                    obj_tool=compile(f, 'a_filename', 'exec')
                    return obj_tool
        except:
            lo.wfile("Error in creating executable tool object......")
tb=ToolsBuilder()

class ToolsVectorGenerator:

  def generate(self,txn,res_set={}):

    v1=[]
    for asset_id in res_set.iterkeys():
        try:
            obj_tools=[]
            if asset_id:
                print "asset_id..............................",asset_id
                log_date=lo.log_date()
                log=log_date+"::"+" \nVector generation for the asset number...:"+str(asset_id)
                lo.wfile(log)
                vector=[]
                tools_arr=[]
                obj_tools=[]
                for tool in res_set[asset_id]:
                    if tool:
                        print "tool..............",tool
                        temp_tool=tb.build(txn,tool,asset_id)
                        print "temp_tool..........",temp_tool

                        #fetch data of tool setting.....
                        sql_tool_setting="select * from tool_asset_settings where tool_id =%s" %(tool,)
                        txn.execute(sql_tool_setting)
                        result_tool_setting = txn.fetchall()
                        tool_id=result_tool_setting[0][1]
                        t_id=int(tool_id)
                        tool_id_arr=[]
                        tool_id_arr.append(t_id)
                        tool_id_arr.append(result_tool_setting)
                        tool_id_arr.append(temp_tool)
                        tools_arr.append(tool_id_arr)

                #fetch data from asset master
                sql_asset="select asset_name from asset_master where asset_id=%s" %(asset_id,)
                txn.execute(sql_asset)
                result_asset = txn.fetchall()
                vector.append(result_asset)
                vector.append(tools_arr)
        except:
            lo.wfile("\nError in getting asset,please check your database or network connection......")
        tvm.executeVector(vector)
tvg=ToolsVectorGenerator()

class Tool:

  def exectool(self,tool):
      exec tool
      return

  def getResult(self,tool):
      return deferToThread(self.exectool, tool)

to=Tool()

class StateMachine:

  def setPriority(self,txn,tup):
        temp=[]
        arr=[]
        for li in tup:
            sql2="select tool_dependency from tool_asset_settings where tool_id =%s" %(li[1],)
            txn.execute(sql2)
            result12 = txn.fetchall()
            arr=[]
            if result12[0][0]!=None:
                tup12=result12[0][0]
                arr=(li[0],tup12)
    #                print "arr.........",arr
                if arr in tup:
                    print "This element is already exist......."
                else:
                    temp.append(arr)
        temp.extend(tup)
        return tuple(temp)

st=StateMachine()

class ToolsVectorExecutionManager(object):

    def executeVector(self,toolsvector):
        print "toolsvector================>",toolsvector
        if toolsvector:
            for tools in toolsvector[1]:
                if tools[2] != None:
                    to.getResult(tools[2])

tvm=ToolsVectorExecutionManager()

class ToolsToExecuteAnalyzer:

  def __init__(self,dbpool=None):
    self.dbpool = dbpool
    self.loopCall = task.LoopingCall(self.myQuery)

  def start(self):
    print "Started scanner"
    self.loopCall.start(3)

  def stop(self):
    print "Stopping scanner"
    self.loopCall.stop()

  def myQuery(self):

    def interact(txn):
      sql="SELECT tool_asset_id,tool_execute_id FROM tool_to_execute where status='0'"
      txn.execute(sql)
      result=txn.fetchall()
      if result:
            tool_asset_id=tuple([int(e[0]) for e in result])
            tool_execute_id=tuple([int(e[1]) for e in result])
            if len(tool_asset_id)>1:
                sql1="SELECT asset_id,tool_id FROM tool_in_assets WHERE tool_asset_id IN %s"%(tool_asset_id,)
            else:
                sql1="SELECT asset_id,tool_id FROM tool_in_assets WHERE tool_asset_id = (%s)"%(tool_asset_id)
            txn.execute(sql1)
            tup = txn.fetchall()

#dependency check for the selected tool
            asset_tool=st.setPriority(txn,tup)

            log_date=lo.log_date()
            log=log_date+"::priority have been set for the tools......\n"
            lo.wfile(log)

#creating group of asset with their tools
            res={}
            for element in asset_tool:
                if element[0] in res:
                    res[element[0]].append(int(element[1]))
                else:
                    res[int(element[0])] = [int(element[1])]
#Recored deletion from tool_to_execute table
            if res!=None and res.keys()!=[]:
                for asset_id in res.iterkeys():
                    if len(tool_execute_id)>1:
                        sql_del="delete from tool_to_execute where tool_execute_id in %s " %(tool_execute_id,)
                    else:
                        sql_del="delete from tool_to_execute where tool_execute_id = %s" %(tool_execute_id)
                txn.execute(sql_del)

#New Addition of vector
            tvg.generate(txn,res)
#            return res

    d = self.dbpool.runInteraction(interact)
    d.addCallbacks(self.printResult,self.printError)

  def printResult(self,res):

    print "In printResult after generate...."

  def printError(self,error):

    print "Got Error: %r" % error
    error.printTraceback()

ToolsToExecuteAnalyzer()


if __name__ == '__main__':

  from twisted.internet import reactor

  dbpool = adbapi.ConnectionPool("MySQLdb", 'localhost', 'test', 'test','test')
  s = ToolsToExecuteAnalyzer(dbpool)
  reactor.callWhenRunning(s.start)
  reactor.addSystemEventTrigger('before','shutdown',s.stop)
  reactor.run()

这是我的全部代码，我只是想知道有多少线程在运行，这意味着每个工具都有一个新线程？

---

无论如何，感谢Matt的帮助。：）

在Matt的所有建议和帮助之后，我有以下代码正在成功运行：

#!usr/bin/python
# Using the "dbmodule" from the previous example, create a ConnectionPool
from twisted.internet import reactor
from twisted.enterprise import adbapi
from twisted.internet import reactor, defer,threads
from twisted.python.threadpool import ThreadPool
import itertools
from twisted.internet.threads import deferToThread
from twisted.internet import reactor, defer, task
from tools.printTime import *
from tools.getVersion import *
from sh_log import *


concurrent = 30
finished=itertools.count(1)
reactor.suggestThreadPoolSize(concurrent)


#Creating Global Instance variables
path="tools"
lo=Log()

class ToolsBuilder:

  def build(self,txn,tool,asset_id):
    if tool:
        print "\n"
        try:
            sql="select tool_filename from tool_master where tool_id = %s" %(tool,)
            sql_asset="select asset_url from asset_master where asset_id = %s" %(asset_id,)
            txn.execute(sql_asset)
            asset_url = txn.fetchall()
            log_date=lo.log_date()
            txn.execute(sql)
            result = txn.fetchall()

            log='\n'+log_date+"::"+str(result[0][0])+ " tool object is created......\n"
            lo.wfile(log)
            temp=(path +'/' + str(result[0][0]))

            if result:
                if temp:
                    f=open(temp).read()
                    obj_tool=compile(f, 'a_filename', 'exec')
                    return obj_tool
        except:
            lo.wfile("Error in creating executable tool object......")
tb=ToolsBuilder()

class ToolsVectorGenerator:

  def generate(self,txn,res_set={}):

    v1=[]
    for asset_id in res_set.iterkeys():
        try:
            obj_tools=[]
            if asset_id:
                print "asset_id..............................",asset_id
                log_date=lo.log_date()
                log=log_date+"::"+" \nVector generation for the asset number...:"+str(asset_id)
                lo.wfile(log)
                vector=[]
                tools_arr=[]
                obj_tools=[]
                for tool in res_set[asset_id]:
                    if tool:
                        print "tool..............",tool
                        temp_tool=tb.build(txn,tool,asset_id)
                        print "temp_tool..........",temp_tool

                        #fetch data of tool setting.....
                        sql_tool_setting="select * from tool_asset_settings where tool_id =%s" %(tool,)
                        txn.execute(sql_tool_setting)
                        result_tool_setting = txn.fetchall()
                        tool_id=result_tool_setting[0][1]
                        t_id=int(tool_id)
                        tool_id_arr=[]
                        tool_id_arr.append(t_id)
                        tool_id_arr.append(result_tool_setting)
                        tool_id_arr.append(temp_tool)
                        tools_arr.append(tool_id_arr)

                #fetch data from asset master
                sql_asset="select asset_name from asset_master where asset_id=%s" %(asset_id,)
                txn.execute(sql_asset)
                result_asset = txn.fetchall()
                vector.append(result_asset)
                vector.append(tools_arr)
        except:
            lo.wfile("\nError in getting asset,please check your database or network connection......")
        tvm.executeVector(vector)
tvg=ToolsVectorGenerator()

class Tool:

  def exectool(self,tool):
      exec tool
      return

  def getResult(self,tool):
      return deferToThread(self.exectool, tool)

to=Tool()

class StateMachine:

  def setPriority(self,txn,tup):
        temp=[]
        arr=[]
        for li in tup:
            sql2="select tool_dependency from tool_asset_settings where tool_id =%s" %(li[1],)
            txn.execute(sql2)
            result12 = txn.fetchall()
            arr=[]
            if result12[0][0]!=None:
                tup12=result12[0][0]
                arr=(li[0],tup12)
    #                print "arr.........",arr
                if arr in tup:
                    print "This element is already exist......."
                else:
                    temp.append(arr)
        temp.extend(tup)
        return tuple(temp)

st=StateMachine()

class ToolsVectorExecutionManager(object):

    def executeVector(self,toolsvector):
        print "toolsvector================>",toolsvector
        if toolsvector:
            for tools in toolsvector[1]:
                if tools[2] != None:
                    to.getResult(tools[2])

tvm=ToolsVectorExecutionManager()

class ToolsToExecuteAnalyzer:

  def __init__(self,dbpool=None):
    self.dbpool = dbpool
    self.loopCall = task.LoopingCall(self.myQuery)

  def start(self):
    print "Started scanner"
    self.loopCall.start(3)

  def stop(self):
    print "Stopping scanner"
    self.loopCall.stop()

  def myQuery(self):

    def interact(txn):
      sql="SELECT tool_asset_id,tool_execute_id FROM tool_to_execute where status='0'"
      txn.execute(sql)
      result=txn.fetchall()
      if result:
            tool_asset_id=tuple([int(e[0]) for e in result])
            tool_execute_id=tuple([int(e[1]) for e in result])
            if len(tool_asset_id)>1:
                sql1="SELECT asset_id,tool_id FROM tool_in_assets WHERE tool_asset_id IN %s"%(tool_asset_id,)
            else:
                sql1="SELECT asset_id,tool_id FROM tool_in_assets WHERE tool_asset_id = (%s)"%(tool_asset_id)
            txn.execute(sql1)
            tup = txn.fetchall()

#dependency check for the selected tool
            asset_tool=st.setPriority(txn,tup)

            log_date=lo.log_date()
            log=log_date+"::priority have been set for the tools......\n"
            lo.wfile(log)

#creating group of asset with their tools
            res={}
            for element in asset_tool:
                if element[0] in res:
                    res[element[0]].append(int(element[1]))
                else:
                    res[int(element[0])] = [int(element[1])]
#Recored deletion from tool_to_execute table
            if res!=None and res.keys()!=[]:
                for asset_id in res.iterkeys():
                    if len(tool_execute_id)>1:
                        sql_del="delete from tool_to_execute where tool_execute_id in %s " %(tool_execute_id,)
                    else:
                        sql_del="delete from tool_to_execute where tool_execute_id = %s" %(tool_execute_id)
                txn.execute(sql_del)

#New Addition of vector
            tvg.generate(txn,res)
#            return res

    d = self.dbpool.runInteraction(interact)
    d.addCallbacks(self.printResult,self.printError)

  def printResult(self,res):

    print "In printResult after generate...."

  def printError(self,error):

    print "Got Error: %r" % error
    error.printTraceback()

ToolsToExecuteAnalyzer()


if __name__ == '__main__':

  from twisted.internet import reactor

  dbpool = adbapi.ConnectionPool("MySQLdb", 'localhost', 'test', 'test','test')
  s = ToolsToExecuteAnalyzer(dbpool)
  reactor.callWhenRunning(s.start)
  reactor.addSystemEventTrigger('before','shutdown',s.stop)
  reactor.run()

这是我的全部代码，我只是想知道有多少线程在运行，这意味着每个工具都有一个新线程？

---

无论如何，谢谢Matt的帮助。：）

您可能还想看看这个片段，它提供了一个ConnectionPool子类，可以在“MySQL服务器已经消失”上重新连接

---

您可能还想看看这个代码段，它提供了一个ConnectionPool子类，可在“MySQL服务器已消失”上重新连接

---

我没有足够的业力来编辑这个问题。是否有人可以编辑此问题并将其格式化为可读版本？谢谢我没有足够的业力来编辑这个问题。是否有人可以编辑此问题并将其格式化为可读版本？谢谢thanx MattH你的解释消除了我的困惑。但我不必运行一个查询，我必须运行两个三个查询，这样一个查询输出就是另一个的输入，这就是我使用runInteraction（）方法的原因。我对runInteraction（）方法“MattH”有一个问题，我能否制作一个方法，使其光标在上面的情况下表示“txn”将用于另一个方法？因为我有这样的要求，我必须多次处理数据库。@jitendra:

runInteraction

是在一个线程中的单个事务中处理数据库。这意味着您必须小心传递的函数的作用，因为它可能会变得非线程安全。长期交易通常是一件坏事。你不想挂在“txn”上。ConnectionPool用于管理数据库连接资源，除非您另有发现，否则请假定它提供了合理的性能。它在不同的数据库中有所不同，但通常创建游标要比创建连接便宜。假设我得到了res（res={1:[2,3]，2:[3,4]}），并且我在另一个方法中传递这个值。现在，基于这些值，我必须像..一样与数据库交互。。def printResult（self，res）：other_class_obj.generate（res）class other_class:generate（self，res={}）：for id in res.iterkeys（）：for tool in res[id]：sql_tool_setting=“select*from tool_asset_settings其中tool_id=%s”%（tool，）#这里您还必须运行此查询和其他查询。那么我应该使用runInteraction方法吗？@jitendra:你为什么要使用另一个类？如果您只使用一次选择的结果来选择更多的内容，那么很可能它可以合并到一个SQL查询中。在这种情况下，当您担心所查询的数据可能会在选择之间发生更改时，可以使用事务。可以将查询合并为一个查询，也可以在定义的单个

interactive

函数中执行所有查询。另外，您应该使用数据库API参数化，而不是自己使用字符串格式。Thanx MattH..您的建议有效我合并了代码，问题解决了。：）如何知道有多少线程&哪个线程在为哪个函数运行？twisted中有预定义的方法？thanx MattH，你的解释消除了我的很多困惑。但我不必运行一个查询，我必须运行两个三个查询，这样一个查询输出就是另一个的输入，这就是我使用runInteraction（）方法的原因。我对runInteraction（）方法“MattH”有一个问题，我能否制作一个方法，使其光标在上面的情况下表示“txn”将用于另一个方法？因为我有这样的要求，我必须