Python BigTable&；gRPC-内存泄漏，如何解决？
我在从云函数写入BigTable时遇到问题 机器的内存消耗增加了，所以我怀疑某处有内存泄漏 BigTable设置：Python BigTable&；gRPC-内存泄漏，如何解决？,python,grpc,google-cloud-bigtable,Python,Grpc,Google Cloud Bigtable,我在从云函数写入BigTable时遇到问题 机器的内存消耗增加了，所以我怀疑某处有内存泄漏 BigTable设置： cbt createtable test cbt createfamily test data 最小代码： os.environ[“GOOGLE\u应用程序\u凭据”]=“GOOGLE\u auth.json” 内存消耗： > 34755 ubuntu 20 0 1118552 128968 21952 S 2.3 0.4 0:04.99 python
cbt createtable test
cbt createfamily test data
os.environ[“GOOGLE\u应用程序\u凭据”]=“GOOGLE\u auth.json”> 34755 ubuntu 20 0 1118552 128968 21952 S 2.3 0.4 0:04.99 python3.6
> 34755 ubuntu 20 0 1118552 129844 21952 S 2.0 0.4 0:05.05 python3.6
> 34755 ubuntu 20 0 1118552 129968 21952 S 1.0 0.4 0:05.08 python3.6
> 34755 ubuntu 20 0 1118552 130780 21952 S 1.6 0.4 0:05.13 python3.6
> 34755 ubuntu 20 0 1119116 132212 21952 S 9.2 0.4 0:05.41 python3.6
> 34755 ubuntu 20 0 1119116 134008 21952 S 4.3 0.4 0:05.54 python3.6
> 34755 ubuntu 20 0 1119116 134944 21952 S 2.6 0.4 0:05.62 python3.6
> 34755 ubuntu 20 0 1119116 137924 21952 S 6.6 0.4 0:05.82 python3.6
> 34755 ubuntu 20 0 1119116 138536 21952 S 1.7 0.4 0:05.87 python3.6
> 34755 ubuntu 20 0 1119116 139988 21952 S 3.0 0.4 0:05.96 python3.6
> 34755 ubuntu 20 0 1119096 141768 21952 S 11.2 0.4 0:06.30 python3.6
> 34755 ubuntu 20 0 1119096 144140 21952 S 4.6 0.4 0:06.44 python3.6
> 34755 ubuntu 20 0 1119096 145140 21952 S 2.6 0.4 0:06.52 python3.6
> 34755 ubuntu 20 0 1119096 148648 21952 S 7.6 0.5 0:06.75 python3.6
> 34755 ubuntu 20 0 1119096 148920 21952 S 0.7 0.5 0:06.77 python3.6
> 34755 ubuntu 20 0 1119096 149828 21952 S 2.3 0.5 0:06.84 python3.6
> 34755 ubuntu 20 0 1119608 152796 21952 S 12.2 0.5 0:07.21 python3.6
> 34755 ubuntu 20 0 1119608 154692 21952 S 4.6 0.5 0:07.35 python3.6
> 34755 ubuntu 20 0 1119608 155448 21952 S 1.6 0.5 0:07.40 python3.6
> 34755 ubuntu 20 0 1119608 156656 21952 S 2.6 0.5 0:07.48 python3.6
> 34755 ubuntu 20 0 1119608 157724 21952 S 2.6 0.5 0:07.56 python3.6
> 34755 ubuntu 20 0 1119608 160796 21952 S 6.9 0.5 0:07.77 python3.6
> 34755 ubuntu 20 0 1119608 161616 21952 S 1.7 0.5 0:07.82 python3.6
> 34755 ubuntu 20 0 1185468 163896 21952 S 12.8 0.5 0:08.21 python3.6
> 34755 ubuntu 20 0 1185476 167612 21952 S 7.6 0.5 0:08.44 python3.6
> 34755 ubuntu 20 0 1185476 170552 21952 S 6.6 0.5 0:08.64 python3.6
> 34755 ubuntu 20 0 1185804 173848 21952 S 13.5 0.5 0:09.05 python3.6
> 34755 ubuntu 20 0 1185804 173848 21952 S 0.3 0.5 0:09.06 python3.6
> 34755 ubuntu 20 0 1185804 176724 21952 S 6.6 0.5 0:09.26 python3.6
> 34755 ubuntu 20 0 1185804 180588 21952 S 8.5 0.6 0:09.52 python3.6
> 34755 ubuntu 20 0 1185804 181524 21952 S 2.3 0.6 0:09.59 python3.6
> 34755 ubuntu 20 0 1186116 183228 21952 S 12.5 0.6 0:09.97 python3.6
> 34755 ubuntu 20 0 1186116 183228 21952 S 0.7 0.6 0:09.99 python3.6
> 34755 ubuntu 20 0 1186116 183276 21952 S 0.3 0.6 0:10.00 python3.6
> 34755 ubuntu 20 0 1186116 183296 21952 S 0.3 0.6 0:10.01 python3.6
> 34755 ubuntu 20 0 1186116 183332 21952 S 0.3 0.6 0:10.02 python3.6
> 34755 ubuntu 20 0 1186116 183436 21952 S 0.7 0.6 0:10.04 python3.6
> 34755 ubuntu 20 0 1186116 183516 21952 S 0.7 0.6 0:10.06 python3.6
> 34755 ubuntu 20 0 1186116 183616 21952 S 0.3 0.6 0:10.07 python3.6
> 34755 ubuntu 20 0 1186116 183656 21952 S 0.3 0.6 0:10.08 python3.6
> 34755 ubuntu 20 0 1186116 183656 21952 S 0.3 0.6 0:10.09 python3.6
> 34755 ubuntu 20 0 1186116 183680 21952 S 0.3 0.6 0:10.10 python3.6
> 34755 ubuntu 20 0 1186116 183708 21952 S 0.3 0.6 0:10.11 python3.6
> 34755 ubuntu 20 0 1186116 183728 21952 S 0.3 0.6 0:10.12 python3.6
---------------------------------------------------------------------------
_Rendezvous Traceback (most recent call last)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/api_core/grpc_helpers.py in next(self)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/grpc/_channel.py in __next__(self)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/grpc/_channel.py in _next(self)
_Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Getting metadata from plugin failed with error: [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]"
debug_error_string = "{"created":"@1548443075.831642536","description":"Getting metadata from plugin failed with error: [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]","file":"src/core/lib/security/credentials/plugin/plugin_credentials.cc","file_line":82,"grpc_status":14}"
>
The above exception was the direct cause of the following exception:
ServiceUnavailable Traceback (most recent call last)
<ipython-input-3-9f1f73192607> in <module>
12 row_key=f"{random.randint(0, sys.maxsize)}_{datetime.datetime.utcnow().strftime('%Y_%m_%d_%H_%M_%S_%f')}",
13 column_family_id="data",
---> 14 data=data
15 )
16
<ipython-input-1-e79f91f922f9> in insert_row(project_id, instance, table, row_key, column_family_id, data)
34 )
35 rows.append(row)
---> 36 table_mutation = table.mutate_rows(rows, retry=Retry(deadline=1200))
37
38 # If the code is not 0, then there is an issue
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/cloud/bigtable/table.py in mutate_rows(self, rows, retry)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/cloud/bigtable/table.py in __call__(self, retry)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/api_core/retry.py in retry_wrapped_func(*args, **kwargs)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/api_core/retry.py in retry_target(target, predicate, sleep_generator, deadline, on_error)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/cloud/bigtable/table.py in _do_mutate_retryable_rows(self)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/api_core/grpc_helpers.py in next(self)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/six.py in raise_from(value, from_value)
ServiceUnavailable: 503 Getting metadata from plugin failed with error: [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
---------------------------------------------------------------------------
_会合追踪（最近一次呼叫最后一次）
~/python3\u virtualenv/python3\u env/lib/python3.6/site-packages/google/api\u core/grpc\u helpers.py在next（self）中
~/python3\u virtualenv/python3\u env/lib/python3.6/site-packages/grpc//u channel.py in\uuuuu next\uuuuu（self）
~/python3\u virtualenv/python3\u env/lib/python3.6/site-packages/grpc//u channel.py in\u next（self）
_会合地点：
上述异常是以下异常的直接原因：
ServiceUnavailable回溯（最近一次呼叫上次）
在里面
12行\u key=f“{random.randint（0，sys.maxsize）}{datetime.datetime.utcnow（）.strftime（“%Y\u%m\u%d\u%H\u%m\u%S\u%f”）”，
13列\u family\u id=“数据”，
--->14数据=数据
15 )
16
插入行中（项目id、实例、表、行键、列族id、数据）
34 )
35行。追加（行）
--->36 table_mutation=table.mutate_行（行，重试=重试（截止日期=1200））
37
38#如果代码不是0，则存在问题
变异行中的~/python3\u virtualenv/python3\u env/lib/python3.6/site-packages/google/cloud/bigtable/table.py（self，rows，重试）
~/python3\u virtualenv/python3\u env/lib/python3.6/site-packages/google/cloud/bigtable/table.py在调用中（self，重试）
函数中的~/python3\u virtualenv/python3\u env/lib/python3.6/site-packages/google/api\u core/retry.py（*args，**kwargs）
重试目标中的~/python3\u virtualenv/python3\u env/lib/python3.6/site-packages/google/api\u core/retry.py（目标、谓词、睡眠生成器、截止日期、打开错误）
~/python3\u virtualenv/python3\u env/lib/python3.6/site-packages/google/cloud/bigtable/table.py in\u do\u mutate\u retryable\u行（self）
~/python3\u virtualenv/python3\u env/lib/python3.6/site-packages/google/api\u core/grpc\u helpers.py在next（self）中
~/python3\u virtualenv/python3\u env/lib/python3.6/site-packages/six.py in raise\u from（value，from\u value）
ServiceUnavailable:503从插件获取元数据失败，错误为：[（'system library'、'fopen'、'打开的文件太多'）、（'BIO routines'、'BIO_new_file'、'system lib'）、（'x509 certificate routines'、'x509_load_cert_crl_file'、'system lib'）]
云函数的一个问题是，所有对象都不是持久的，因此每个写请求都会打开客户端，但由于gRPC客户端没有释放，因此在该实例下仍会使用内存消耗。但是，由于gRPC客户端不关闭，是否有办法强制关闭BigTable下使用的gRPC客户端？您应该能够使用以下代码关闭客户端。它是在pull请求中引入的
Github上有一个相关的问题：.@SolomonDuskis您可以在调用之间重用BigTable客户机，但您不知道客户机何时被清理。所以，当客户端消失时，您需要检查并创建客户端。这是一个平淡无奇的解决方案，因为现在很难进行单元测试。我认为更好的解决方案是关闭客户端并释放资源。不管怎样，如果将客户端创建拉到insert_row函数之外，您可能会看到更好的性能。传入client_bigtable将是一个更安全的选择。客户端应该为您处理所有基础grpc连接，而不需要为每个请求重新创建。
> 34755 ubuntu 20 0 1118552 128968 21952 S 2.3 0.4 0:04.99 python3.6
> 34755 ubuntu 20 0 1118552 129844 21952 S 2.0 0.4 0:05.05 python3.6
> 34755 ubuntu 20 0 1118552 129968 21952 S 1.0 0.4 0:05.08 python3.6
> 34755 ubuntu 20 0 1118552 130780 21952 S 1.6 0.4 0:05.13 python3.6
> 34755 ubuntu 20 0 1119116 132212 21952 S 9.2 0.4 0:05.41 python3.6
> 34755 ubuntu 20 0 1119116 134008 21952 S 4.3 0.4 0:05.54 python3.6
> 34755 ubuntu 20 0 1119116 134944 21952 S 2.6 0.4 0:05.62 python3.6
> 34755 ubuntu 20 0 1119116 137924 21952 S 6.6 0.4 0:05.82 python3.6
> 34755 ubuntu 20 0 1119116 138536 21952 S 1.7 0.4 0:05.87 python3.6
> 34755 ubuntu 20 0 1119116 139988 21952 S 3.0 0.4 0:05.96 python3.6
> 34755 ubuntu 20 0 1119096 141768 21952 S 11.2 0.4 0:06.30 python3.6
> 34755 ubuntu 20 0 1119096 144140 21952 S 4.6 0.4 0:06.44 python3.6
> 34755 ubuntu 20 0 1119096 145140 21952 S 2.6 0.4 0:06.52 python3.6
> 34755 ubuntu 20 0 1119096 148648 21952 S 7.6 0.5 0:06.75 python3.6
> 34755 ubuntu 20 0 1119096 148920 21952 S 0.7 0.5 0:06.77 python3.6
> 34755 ubuntu 20 0 1119096 149828 21952 S 2.3 0.5 0:06.84 python3.6
> 34755 ubuntu 20 0 1119608 152796 21952 S 12.2 0.5 0:07.21 python3.6
> 34755 ubuntu 20 0 1119608 154692 21952 S 4.6 0.5 0:07.35 python3.6
> 34755 ubuntu 20 0 1119608 155448 21952 S 1.6 0.5 0:07.40 python3.6
> 34755 ubuntu 20 0 1119608 156656 21952 S 2.6 0.5 0:07.48 python3.6
> 34755 ubuntu 20 0 1119608 157724 21952 S 2.6 0.5 0:07.56 python3.6
> 34755 ubuntu 20 0 1119608 160796 21952 S 6.9 0.5 0:07.77 python3.6
> 34755 ubuntu 20 0 1119608 161616 21952 S 1.7 0.5 0:07.82 python3.6
> 34755 ubuntu 20 0 1185468 163896 21952 S 12.8 0.5 0:08.21 python3.6
> 34755 ubuntu 20 0 1185476 167612 21952 S 7.6 0.5 0:08.44 python3.6
> 34755 ubuntu 20 0 1185476 170552 21952 S 6.6 0.5 0:08.64 python3.6
> 34755 ubuntu 20 0 1185804 173848 21952 S 13.5 0.5 0:09.05 python3.6
> 34755 ubuntu 20 0 1185804 173848 21952 S 0.3 0.5 0:09.06 python3.6
> 34755 ubuntu 20 0 1185804 176724 21952 S 6.6 0.5 0:09.26 python3.6
> 34755 ubuntu 20 0 1185804 180588 21952 S 8.5 0.6 0:09.52 python3.6
> 34755 ubuntu 20 0 1185804 181524 21952 S 2.3 0.6 0:09.59 python3.6
> 34755 ubuntu 20 0 1186116 183228 21952 S 12.5 0.6 0:09.97 python3.6
> 34755 ubuntu 20 0 1186116 183228 21952 S 0.7 0.6 0:09.99 python3.6
> 34755 ubuntu 20 0 1186116 183276 21952 S 0.3 0.6 0:10.00 python3.6
> 34755 ubuntu 20 0 1186116 183296 21952 S 0.3 0.6 0:10.01 python3.6
> 34755 ubuntu 20 0 1186116 183332 21952 S 0.3 0.6 0:10.02 python3.6
> 34755 ubuntu 20 0 1186116 183436 21952 S 0.7 0.6 0:10.04 python3.6
> 34755 ubuntu 20 0 1186116 183516 21952 S 0.7 0.6 0:10.06 python3.6
> 34755 ubuntu 20 0 1186116 183616 21952 S 0.3 0.6 0:10.07 python3.6
> 34755 ubuntu 20 0 1186116 183656 21952 S 0.3 0.6 0:10.08 python3.6
> 34755 ubuntu 20 0 1186116 183656 21952 S 0.3 0.6 0:10.09 python3.6
> 34755 ubuntu 20 0 1186116 183680 21952 S 0.3 0.6 0:10.10 python3.6
> 34755 ubuntu 20 0 1186116 183708 21952 S 0.3 0.6 0:10.11 python3.6
> 34755 ubuntu 20 0 1186116 183728 21952 S 0.3 0.6 0:10.12 python3.6
---------------------------------------------------------------------------
_Rendezvous Traceback (most recent call last)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/api_core/grpc_helpers.py in next(self)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/grpc/_channel.py in __next__(self)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/grpc/_channel.py in _next(self)
_Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Getting metadata from plugin failed with error: [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]"
debug_error_string = "{"created":"@1548443075.831642536","description":"Getting metadata from plugin failed with error: [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]","file":"src/core/lib/security/credentials/plugin/plugin_credentials.cc","file_line":82,"grpc_status":14}"
>
The above exception was the direct cause of the following exception:
ServiceUnavailable Traceback (most recent call last)
<ipython-input-3-9f1f73192607> in <module>
12 row_key=f"{random.randint(0, sys.maxsize)}_{datetime.datetime.utcnow().strftime('%Y_%m_%d_%H_%M_%S_%f')}",
13 column_family_id="data",
---> 14 data=data
15 )
16
<ipython-input-1-e79f91f922f9> in insert_row(project_id, instance, table, row_key, column_family_id, data)
34 )
35 rows.append(row)
---> 36 table_mutation = table.mutate_rows(rows, retry=Retry(deadline=1200))
37
38 # If the code is not 0, then there is an issue
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/cloud/bigtable/table.py in mutate_rows(self, rows, retry)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/cloud/bigtable/table.py in __call__(self, retry)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/api_core/retry.py in retry_wrapped_func(*args, **kwargs)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/api_core/retry.py in retry_target(target, predicate, sleep_generator, deadline, on_error)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/cloud/bigtable/table.py in _do_mutate_retryable_rows(self)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/google/api_core/grpc_helpers.py in next(self)
~/python3_virtualenv/python3_env/lib/python3.6/site-packages/six.py in raise_from(value, from_value)
ServiceUnavailable: 503 Getting metadata from plugin failed with error: [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
client.table_data_client.transport.channel.close()