Python AWS lambda函数实现中的Concurrent.futures
我正在使用AWS Lambda python函数将EBS/RDS快照复制到另一个区域以进行灾难恢复。 我遇到的问题是一次只能复制5个快照。 如果我当时试图复制超过5个,我将得到一个错误:Python AWS lambda函数实现中的Concurrent.futures,python,amazon-web-services,aws-lambda,boto3,concurrent.futures,Python,Amazon Web Services,Aws Lambda,Boto3,Concurrent.futures,我正在使用AWS Lambda python函数将EBS/RDS快照复制到另一个区域以进行灾难恢复。 我遇到的问题是一次只能复制5个快照。 如果我当时试图复制超过5个,我将得到一个错误: botocore.exceptions.ClientError: An error occurred (ResourceLimitExceeded) when calling the CopySnapshot operation: Too many snapshot copies in progress. Th
botocore.exceptions.ClientError: An error occurred (ResourceLimitExceeded) when calling the CopySnapshot operation: Too many snapshot copies in progress. The limit is 5 for this destination region.
为了避免这种情况,我添加了一个服务员函数,用于检查目标区域中快照的状态,并在快照完成状态后继续循环。
它工作得很好,但在本例中,它一次只能处理一个快照。
问题是,如何为同时复制5个快照的并行任务实现concurrent.futures模块
waiter = client_ec2_dst.get_waiter('snapshot_completed')
message = ""
for i in ec2_snapshots_src:
# snapshot_tags_filtered = ([item for item in i["Tags"] if item['Key'] != 'aws:backup:source-resource']
# snapshot_tags_filtered.append({'Key': 'delete_On', 'Value': delete_on})
# snapshot_tags_filtered.append({'Key': 'src_Id', 'Value': i["SnapshotId"]})
try:
response = client_ec2_dst.copy_snapshot(
Description='[Disaster Recovery] copied from us-east-1',
SourceRegion=region_src,
SourceSnapshotId=i["SnapshotId"],
DryRun=False,
# Encrypted=True,
# KmsKeyId='1e287363-89f6-4837-a619-b550ff28c211',
)
new_snapshot_id = response["SnapshotId"]
waiter.wait(
SnapshotIds=[new_snapshot_id],
WaiterConfig={'Delay': 5, 'MaxAttempts': 120}
)
snapshot_src_name = ([dic['Value'] for dic in snapshot_tags_filtered if dic['Key'] == 'Name'])
message += ("Started copying latest EBS snapshot: " + i["SnapshotId"] + " for EC2 instance: " + str(snapshot_src_name) + " from: " + region_src + " to: " + region_dst + " with new id: " + new_snapshot_id + ".\n")
# Adding tags to snapshots in destination region
tag_src = [new_snapshot_id]
tag = client_ec2_dst.create_tags(
DryRun=False,
Resources=tag_src,
Tags=snapshot_tags_filtered
)
except Exception as e:
raise e
您可以使用concurrent executor和
max_workers
参数来限制同时运行的作业数量。像这样:
import concurrent.futures
def copy_snapshot(snapshot_id):
waiter = client_ec2_dst.get_waiter('snapshot_completed')
response = client_ec2_dst.copy_snapshot(
Description='[Disaster Recovery] copied from us-east-1',
SourceRegion=region_src,
SourceSnapshotId=snapshot_id,
DryRun=False
)
new_snapshot_id = response["SnapshotId"]
waiter.wait(
SnapshotIds=[new_snapshot_id],
WaiterConfig={'Delay': 5, 'MaxAttempts': 120}
)
# Copy snapshots in parallel, but no more than 5 at a time:
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
futures = [
executor.submit(copy_snapshot, s['SnapshotId'])
for s in ec2_snapshots_src]
for future in futures:
future.result()
您可以使用concurrent executor和
max_workers
参数来限制同时运行的作业数量。像这样:
import concurrent.futures
def copy_snapshot(snapshot_id):
waiter = client_ec2_dst.get_waiter('snapshot_completed')
response = client_ec2_dst.copy_snapshot(
Description='[Disaster Recovery] copied from us-east-1',
SourceRegion=region_src,
SourceSnapshotId=snapshot_id,
DryRun=False
)
new_snapshot_id = response["SnapshotId"]
waiter.wait(
SnapshotIds=[new_snapshot_id],
WaiterConfig={'Delay': 5, 'MaxAttempts': 120}
)
# Copy snapshots in parallel, but no more than 5 at a time:
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
futures = [
executor.submit(copy_snapshot, s['SnapshotId'])
for s in ec2_snapshots_src]
for future in futures:
future.result()
我更新了问题描述中的代码,因为我传递了snapshot\u tags\u filtered变量并在循环中生成了一条消息。我不知道如何实现您的解决方案。您可以将此新逻辑放到
copy\u snapshot
。这个新逻辑需要标签列表——它们可以作为第二个参数传递。或者,它可以是单个参数,它是一个字典(来自ec2\u snapshots\u src
的单个项目),现在我在前面的函数中修改标记,所以我可以将它传递到copy\u snapshot
函数中,谢谢。但是如何收集每次函数执行时生成的消息
数据开关?您可以从函数返回它们,并在以后每次调用.result()
来收集返回值。如下所示:messages=[f.result()表示未来中的f]
我更新问题描述中的代码,因为我传递了snapshot\u tags\u filtered变量并在循环中生成一条消息。我不知道如何实现您的解决方案。您可以将此新逻辑放到copy\u snapshot
。这个新逻辑需要标签列表——它们可以作为第二个参数传递。或者,它可以是单个参数,它是一个字典(来自ec2\u snapshots\u src
的单个项目),现在我在前面的函数中修改标记,所以我可以将它传递到copy\u snapshot
函数中,谢谢。但是如何收集每次函数执行时生成的消息
数据开关?您可以从函数返回它们,并在以后每次调用.result()
来收集返回值。如下所示:messages=[f.result()表示期货中的f]