如何分析Python脚本？
而其他的编码竞赛通常有最长的运行时间，或者人们吹嘘他们的特定解决方案运行得有多快。在Python中，有时这些方法有点笨拙——例如，将计时代码添加到如何分析Python脚本？,python,performance,optimization,time-complexity,profiling,Python,Performance,Optimization,Time Complexity,Profiling,而其他的编码竞赛通常有最长的运行时间，或者人们吹嘘他们的特定解决方案运行得有多快。在Python中，有时这些方法有点笨拙——例如，将计时代码添加到\uuuuu main\uuuu 分析Python程序运行所需时间的好方法是什么？Python包含一个名为。它不仅给出了总运行时间，还分别给出了每个函数的时间，并告诉您每个函数被调用了多少次，从而很容易确定应该在哪里进行优化 您可以在代码中或从解释器中调用它，如下所示： import cProfile cProfile.run('foo()') im
\uuuuu main\uuuu分析Python程序运行所需时间的好方法是什么？Python包含一个名为。它不仅给出了总运行时间，还分别给出了每个函数的时间，并告诉您每个函数被调用了多少次，从而很容易确定应该在哪里进行优化 您可以在代码中或从解释器中调用它，如下所示：
import cProfile
cProfile.run('foo()')
import statprof
with statprof.profile():
my_questionable_function()
python lsprofcalltree.py -o callgrind.1 test.py
python -m cProfile myscript.py
python -m cProfile %1
profile euler048.py
0.061CPU秒内调用1007个函数
订购人：标准名称
ncalls tottime percall cumtime percall文件名：lineno（函数）
1 0.000 0.000 0.061 0.061 :1()
1000 0.051 0.000 0.051 0.000 euler048.py:2（）
1 0.005 0.005 0.061 0.061 euler048.py:2（）
1 0.000 0.000 0.061 0.061{execfile}
1 0.002 0.002 0.053 0.053{map}
1 0.000 0.000 0.000 0.000{方法'disable'of'\u lsprof.Profiler objects}
1 0.000 0.000 0.000 0.000{范围}
1 0.003 0.003 0.003 0.003{sum}
在Virtaal中有一个非常有用的类和装饰器，它可以使分析（甚至对于特定的方法/函数）变得非常简单。然后可以在KCacheGrind中轻松地查看输出。值得指出的是，使用探查器仅在主线程上工作（默认情况下），如果使用它们，您将无法从其他线程获得任何信息。这可能有点牵强附会，因为它在本文中完全没有提及 如果您还想评测线程，那么您需要查看文档中的 您还可以创建自己的
threading.Threadclass ProfiledThread(threading.Thread):
# Overrides threading.Thread.run()
def run(self):
profiler = cProfile.Profile()
try:
return profiler.runcall(threading.Thread.run, self)
finally:
profiler.dump_stats('myprofile-%d.profile' % (self.ident,))
并使用
ProfiledThreadpython -m cProfile -s time mine.py <args>
安装：
pip install pycallgraph
不幸的是，我经常 点：图形对于开罗渲染器位图太大。缩放0.257079以适合 这让我的图像变得非常小。因此，我通常创建svg文件：
pycallgraph -f svg -o pycallgraph.svg mine.py <args>
pip install gprof2dot
python -m cProfile -o profile.pstats mine.py
gprof2dot -f pstats profile.pstats | dot -Tsvg -o mine.svg
一个不错的评测模块是line_profiler（使用脚本kernprof.py调用）。可以下载
我的理解是cProfile只提供了每个函数花费的总时间的信息。因此，单独的代码行不会计时。这是科学计算中的一个问题，因为通常一行代码会花费很多时间。另外，我记得，cProfile没有赶上我花在say numpy.dot上的时间。根据Joe Shaw关于多线程代码无法按预期工作的回答，我认为cProfile中的
runcallself.enable（）self.disable（）pip安装pycallgraphpycallgraph graphviz -- ./mypythonscript.py
from pycallgraph import PyCallGraph
from pycallgraph.output import GraphvizOutput
with PyCallGraph(output=GraphvizOutput()):
code_to_profile()
pycallgraph.png$ sudo apt-get install graphviz
$ git clone https://github.com/jrfonseca/gprof2dot
$ ln -s "$PWD"/gprof2dot/gprof2dot.py ~/bin
$ cd $PROJECT_DIR
$ gprof2dot.py -f pstats profile.pstats | dot -Tsvg -o callgraph.svg
@staticmethod
def startProfiler():
yappi.start()
@staticmethod
def stopProfiler():
yappi.stop()
@staticmethod
def printProfiler():
stats = yappi.get_stats(yappi.SORTTYPE_TTOT, yappi.SORTORDER_DESC, 20)
statPrint = '\n'
namesArr = [len(str(stat[0])) for stat in stats.func_stats]
log.debug("namesArr %s", str(namesArr))
maxNameLen = max(namesArr)
log.debug("maxNameLen: %s", maxNameLen)
for stat in stats.func_stats:
nameAppendSpaces = [' ' for i in range(maxNameLen - len(stat[0]))]
log.debug('nameAppendSpaces: %s', nameAppendSpaces)
blankSpace = ''
for space in nameAppendSpaces:
blankSpace += space
log.debug("adding spaces: %s", len(nameAppendSpaces))
statPrint = statPrint + str(stat[0]) + blankSpace + " " + str(stat[1]).ljust(8) + "\t" + str(
round(stat[2], 2)).ljust(8 - len(str(stat[2]))) + "\t" + str(round(stat[3], 2)) + "\n"
log.log(1000, "\nname" + ''.ljust(maxNameLen - 4) + " ncall \tttot \ttsub")
log.log(1000, statPrint)
startProfilerprintProfiler2014-02-19 16:32:24,128-|SVR-MAIN |-(Thread-3 )-Level 1000:
name ncall ttot tsub
2014-02-19 16:32:24,128-|SVR-MAIN |-(Thread-3 )-Level 1000:
C:\Python27\lib\sched.py.run:80 22 0.11 0.05
M:\02_documents\_repos\09_aheadRepos\apps\ahdModbusSrv\pyAheadRpcSrv\xmlRpc.py.iterFnc:293 22 0.11 0.0
M:\02_documents\_repos\09_aheadRepos\apps\ahdModbusSrv\serverMain.py.makeIteration:515 22 0.11 0.0
M:\02_documents\_repos\09_aheadRepos\apps\ahdModbusSrv\pyAheadRpcSrv\PicklingXMLRPC.py._dispatch:66 1 0.0 0.0
C:\Python27\lib\BaseHTTPServer.py.date_time_string:464 1 0.0 0.0
c:\users\zasiec~1\appdata\local\temp\easy_install-hwcsr1\psutil-1.1.2-py2.7-win32.egg.tmp\psutil\_psmswindows.py._get_raw_meminfo:243 4 0.0 0.0
C:\Python27\lib\SimpleXMLRPCServer.py.decode_request_content:537 1 0.0 0.0
c:\users\zasiec~1\appdata\local\temp\easy_install-hwcsr1\psutil-1.1.2-py2.7-win32.egg.tmp\psutil\_psmswindows.py.get_system_cpu_times:148 4 0.0 0.0
<string>.__new__:8 220 0.0 0.0
C:\Python27\lib\socket.py.close:276 4 0.0 0.0
C:\Python27\lib\threading.py.__init__:558 1 0.0 0.0
<string>.__new__:8 4 0.0 0.0
C:\Python27\lib\threading.py.notify:372 1 0.0 0.0
C:\Python27\lib\rfc822.py.getheader:285 4 0.0 0.0
C:\Python27\lib\BaseHTTPServer.py.handle_one_request:301 1 0.0 0.0
C:\Python27\lib\xmlrpclib.py.end:816 3 0.0 0.0
C:\Python27\lib\SimpleXMLRPCServer.py.do_POST:467 1 0.0 0.0
C:\Python27\lib\SimpleXMLRPCServer.py.is_rpc_path_valid:460 1 0.0 0.0
C:\Python27\lib\SocketServer.py.close_request:475 1 0.0 0.0
c:\users\zasiec~1\appdata\local\temp\easy_install-hwcsr1\psutil-1.1.2-py2.7-win32.egg.tmp\psutil\__init__.py.cpu_times:1066 4 0.0 0.0
同样值得一提的是GUI cProfile转储查看器。它允许您排序和选择，从而放大程序的相关部分。图片中矩形的大小与所用时间成正比。如果将鼠标悬停在一个矩形上，它会高亮显示表中的调用和屏幕上的所有位置
$ sudo apt-get install graphviz
$ git clone https://github.com/jrfonseca/gprof2dot
$ ln -s "$PWD"/gprof2dot/gprof2dot.py ~/bin
$ cd $PROJECT_DIR
$ gprof2dot.py -f pstats profile.pstats | dot -Tsvg -o callgraph.svg
@staticmethod
def startProfiler():
yappi.start()
@staticmethod
def stopProfiler():
yappi.stop()
@staticmethod
def printProfiler():
stats = yappi.get_stats(yappi.SORTTYPE_TTOT, yappi.SORTORDER_DESC, 20)
statPrint = '\n'
namesArr = [len(str(stat[0])) for stat in stats.func_stats]
log.debug("namesArr %s", str(namesArr))
maxNameLen = max(namesArr)
log.debug("maxNameLen: %s", maxNameLen)
for stat in stats.func_stats:
nameAppendSpaces = [' ' for i in range(maxNameLen - len(stat[0]))]
log.debug('nameAppendSpaces: %s', nameAppendSpaces)
blankSpace = ''
for space in nameAppendSpaces:
blankSpace += space
log.debug("adding spaces: %s", len(nameAppendSpaces))
statPrint = statPrint + str(stat[0]) + blankSpace + " " + str(stat[1]).ljust(8) + "\t" + str(
round(stat[2], 2)).ljust(8 - len(str(stat[2]))) + "\t" + str(round(stat[3], 2)) + "\n"
log.log(1000, "\nname" + ''.ljust(maxNameLen - 4) + " ncall \tttot \ttsub")
log.log(1000, statPrint)
2014-02-19 16:32:24,128-|SVR-MAIN |-(Thread-3 )-Level 1000:
name ncall ttot tsub
2014-02-19 16:32:24,128-|SVR-MAIN |-(Thread-3 )-Level 1000:
C:\Python27\lib\sched.py.run:80 22 0.11 0.05
M:\02_documents\_repos\09_aheadRepos\apps\ahdModbusSrv\pyAheadRpcSrv\xmlRpc.py.iterFnc:293 22 0.11 0.0
M:\02_documents\_repos\09_aheadRepos\apps\ahdModbusSrv\serverMain.py.makeIteration:515 22 0.11 0.0
M:\02_documents\_repos\09_aheadRepos\apps\ahdModbusSrv\pyAheadRpcSrv\PicklingXMLRPC.py._dispatch:66 1 0.0 0.0
C:\Python27\lib\BaseHTTPServer.py.date_time_string:464 1 0.0 0.0
c:\users\zasiec~1\appdata\local\temp\easy_install-hwcsr1\psutil-1.1.2-py2.7-win32.egg.tmp\psutil\_psmswindows.py._get_raw_meminfo:243 4 0.0 0.0
C:\Python27\lib\SimpleXMLRPCServer.py.decode_request_content:537 1 0.0 0.0
c:\users\zasiec~1\appdata\local\temp\easy_install-hwcsr1\psutil-1.1.2-py2.7-win32.egg.tmp\psutil\_psmswindows.py.get_system_cpu_times:148 4 0.0 0.0
<string>.__new__:8 220 0.0 0.0
C:\Python27\lib\socket.py.close:276 4 0.0 0.0
C:\Python27\lib\threading.py.__init__:558 1 0.0 0.0
<string>.__new__:8 4 0.0 0.0
C:\Python27\lib\threading.py.notify:372 1 0.0 0.0
C:\Python27\lib\rfc822.py.getheader:285 4 0.0 0.0
C:\Python27\lib\BaseHTTPServer.py.handle_one_request:301 1 0.0 0.0
C:\Python27\lib\xmlrpclib.py.end:816 3 0.0 0.0
C:\Python27\lib\SimpleXMLRPCServer.py.do_POST:467 1 0.0 0.0
C:\Python27\lib\SimpleXMLRPCServer.py.is_rpc_path_valid:460 1 0.0 0.0
C:\Python27\lib\SocketServer.py.close_request:475 1 0.0 0.0
c:\users\zasiec~1\appdata\local\temp\easy_install-hwcsr1\psutil-1.1.2-py2.7-win32.egg.tmp\psutil\__init__.py.cpu_times:1066 4 0.0 0.0
@staticmethod
def printProfile():
yappi.get_func_stats().print_all()
$ python -m cprofilev /your/python/program
# Go to http://localhost:4000 to view collected statistics.
import cProfile
cProfile.runctx('foo()', None, locals())
def count():
from math import sqrt
for x in range(10**5):
sqrt(x)
if __name__ == '__main__':
import cProfile, pstats
cProfile.run("count()", "{}.profile".format(__file__))
s = pstats.Stats("{}.profile".format(__file__))
s.strip_dirs()
s.sort_stats("time").print_stats(10)
pip install git+git://github.com/bos/statprof.py@1a33eba91899afe17a8b752c6dfdec6f05dd0c01
import statprof
with statprof.profile():
my_questionable_function()
python -m cProfile -o script.profile script.py
pyprof2calltree -i script.profile -o script.calltree
kcachegrind script.calltree
apt-get install kcachegrind
pip install pyprof2calltree
python lsprofcalltree.py -o callgrind.1 test.py
time python python_prog.py
1. pip install snakeviz
2. python -m cProfile -o temp.dat <PROGRAM>.py
3. snakeviz temp.dat
pip install tuna
python3 -m cProfile -o program.prof yourfile.py
python3 -X importprofile yourfile.py 2> import.log
tuna program.prof
pip install gprof2dot_magic
%load_ext gprof2dot_magic
%gprof2dot print('hello world')
$ pip install pyinstrument
$ python -m pyinstrument ./prog.py
import cProfile, pstats
class _ProfileFunc:
def __init__(self, func, sort_stats_by):
self.func = func
self.profile_runs = []
self.sort_stats_by = sort_stats_by
def __call__(self, *args, **kwargs):
pr = cProfile.Profile()
pr.enable() # this is the profiling section
retval = self.func(*args, **kwargs)
pr.disable()
self.profile_runs.append(pr)
ps = pstats.Stats(*self.profile_runs).sort_stats(self.sort_stats_by)
return retval, ps
def cumulative_profiler(amount_of_times, sort_stats_by='time'):
def real_decorator(function):
def wrapper(*args, **kwargs):
nonlocal function, amount_of_times, sort_stats_by # for python 2.x remove this row
profiled_func = _ProfileFunc(function, sort_stats_by)
for i in range(amount_of_times):
retval, ps = profiled_func(*args, **kwargs)
ps.print_stats()
return retval # returns the results of the function
return wrapper
if callable(amount_of_times): # incase you don't want to specify the amount of times
func = amount_of_times # amount_of_times is the function in here
amount_of_times = 5 # the default amount
return real_decorator(func)
return real_decorator
import time
@cumulative_profiler
def baz():
time.sleep(1)
time.sleep(2)
return 1
baz()
20 function calls in 15.003 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
10 15.003 1.500 15.003 1.500 {built-in method time.sleep}
5 0.000 0.000 15.003 3.001 <ipython-input-9-c89afe010372>:3(baz)
5 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
@cumulative_profiler(3)
def baz():
...
Install by: pip install auto_profiler
Time [Hits * PerHit] Function name [Called from] [function location]
-----------------------------------------------------------------------
8.974s [1 * 8.974] main [auto-profiler/profiler.py:267] [/test/t2.py:30]
├── 5.954s [5 * 1.191] f1 [/test/t2.py:34] [/test/t2.py:14]
│ └── 5.954s [5 * 1.191] mysleep [/test/t2.py:15] [/test/t2.py:17]
│ └── 5.954s [5 * 1.191] <time.sleep>
|
|
| # The rest is for the example recursive function call fact
└── 3.020s [1 * 3.020] fact [/test/t2.py:36] [/test/t2.py:20]
├── 0.849s [1 * 0.849] f1 [/test/t2.py:21] [/test/t2.py:14]
│ └── 0.849s [1 * 0.849] mysleep [/test/t2.py:15] [/test/t2.py:17]
│ └── 0.849s [1 * 0.849] <time.sleep>
└── 2.171s [1 * 2.171] fact [/test/t2.py:24] [/test/t2.py:20]
├── 1.552s [1 * 1.552] f1 [/test/t2.py:21] [/test/t2.py:14]
│ └── 1.552s [1 * 1.552] mysleep [/test/t2.py:15] [/test/t2.py:17]
└── 0.619s [1 * 0.619] fact [/test/t2.py:24] [/test/t2.py:20]
└── 0.619s [1 * 0.619] f1 [/test/t2.py:21] [/test/t2.py:14]
austin python3 my_script.py
austin python3 my_script.py | flamegraph.pl > my_script_profile.svg