Python脚本由于CPU利用率达到99%而随机失败,内核杀死了我的脚本
我有一个python脚本,它在达到99%的CPU利用率后被内核杀死。python正在对postgresql数据库执行SQL查询,并将结果附加到单个JSON文件中。当我到达查询150,CPU利用率达到99%以上,然后内核终止脚本时,就会出现问题。如何改进我的代码以避免这种情况发生?我将非常感谢你的帮助 代码Python脚本由于CPU利用率达到99%而随机失败,内核杀死了我的脚本,python,sql,python-3.x,postgresql,cpu,Python,Sql,Python 3.x,Postgresql,Cpu,我有一个python脚本,它在达到99%的CPU利用率后被内核杀死。python正在对postgresql数据库执行SQL查询,并将结果附加到单个JSON文件中。当我到达查询150,CPU利用率达到99%以上,然后内核终止脚本时,就会出现问题。如何改进我的代码以避免这种情况发生?我将非常感谢你的帮助 代码 from __future__ import print_function try: import psycopg2 except ImportError: raise Im
from __future__ import print_function
try:
import psycopg2
except ImportError:
raise ImportError('\n\033[33mpsycopg2 library missing. pip install psycopg2\033[1;m\n')
sys.exit(1)
import re
import sys
import json
import pprint
import time
outfilepath = "crtsh_output/crtsh_flat_file"
outfilepath = "crtsh2" + ".json"
DB_HOST = 'crt.sh'
DB_NAME = 'certwatch'
DB_USER = 'guest'
# DELAY = 0
def connect_to_db():
start = 0
offset = 10
flag = True
while flag:
filepath = 'forager.txt'
with open(filepath) as fp, open(outfilepath) as outfile:
unique_domains = ''
try:
conn = psycopg2.connect("dbname={0} user={1} host={2}".format(DB_NAME, DB_USER, DB_HOST))
cursor = conn.cursor()
cursor.itersize = 10000
for cnt, domain_name in enumerate(fp):
print("Line {}: {}".format(cnt, domain_name))
print(domain_name)
domain_name = domain_name.rstrip()
cursor.execute('''SELECT c.id, x509_commonName(c.certificate), x509_issuerName(c.certificate), x509_notBefore(c.certificate), x509_notAfter(c.certificate), x509_issuerName(c.certificate), x509_keyAlgorithm(c.certificate), x509_keySize(c.certificate), x509_publicKeyMD5(c.certificate), x509_publicKey(c.certificate), x509_rsaModulus(c.certificate), x509_serialNumber(c.certificate), x509_signatureHashAlgorithm(c.certificate), x509_signatureKeyAlgorithm(c.certificate), x509_subjectName(c.certificate), x509_name(c.certificate), x509_name_print(c.certificate), x509_commonName(c.certificate), x509_subjectKeyIdentifier(c.certificate), x509_extKeyUsages(c.certificate), x509_certPolicies(c.certificate), x509_canIssueCerts(c.certificate), x509_getPathLenConstraint(c.certificate), x509_altNames(c.certificate), x509_altNames_raw(c.certificate), x509_cRLDistributionPoints(c.certificate), x509_authorityInfoAccess(c.certificate), x509_print(c.certificate), x509_anyNamesWithNULs(c.certificate), x509_extensions(c.certificate), x509_tbscert_strip_ct_ext(c.certificate), x509_hasROCAFingerprint(c.certificate)
FROM certificate c, certificate_identity ci WHERE
c.id= ci.certificate_id AND ci.name_type = 'dNSName' AND lower(ci.name_value) =
lower(%s) AND x509_notAfter(c.certificate) > statement_timestamp()''', (domain_name,))
# query db with start and offset
unique_domains = cursor.fetchall()
if not unique_domains:
flag = False
else:
# do processing with your data
pprint.pprint(unique_domains)
outfile.write(json.dumps(unique_domains, sort_keys=True, indent=4, default=str, ensure_ascii = False))
offset += limit
except Exception as error:
print(str(error))
if __name__ == "__main__":
connect_to_db()
内核日志中的消息是什么?在查询150时,利用率总是达到99%吗?您是否尝试过限制该数字周围的查询数量,以查看是否有新的/有趣的事情发生?@Gigaflop如果我将结果数量限制为1000,则利用率不会达到99%。但是,我想返回所有结果,因此不能限制结果。@LaurenzAlbe感谢您的帮助。我在内核日志中找不到错误消息。我正在登录/etc/rsyslog.d/50-default.conf?是这个吗?如果没有,我应该去哪里找?@bedford取决于您的Linux发行版。询问系统管理员。如果没有日志条目,您如何知道内核杀死了后端?