如何从S3中的130000多个Json文件中快速提取信息？

我有一个S3是超过130k的Json文件，我需要根据Json文件中的数据计算数字，例如计算发言者的性别。我目前正在使用s3 Paginator和JSON.load读取每个文件并从表单中提取信息。但是每秒处理如此大量的2-3个文件需要很长时间。我怎样才能加快这个过程？如果可能，请提供工作代码示例。非常感谢。 以下是我的一些代码：

client = boto3.client('s3')
paginator = client.get_paginator('list_objects_v2')
result = paginator.paginate(Bucket='bucket-name',StartAfter='') 

for page in result:
    if "Contents" in page:
        for key in page[ "Contents" ]:
            keyString = key[ "Key" ]
                s3 = boto3.resource('s3')
                content_object = s3.Bucket('bucket-name').Object(str(keyString))
                    file_content = content_object.get()['Body'].read().decode('utf-8')
                    json_content = json.loads(file_content)
                    x = (json_content['dict-name'])

---

为了使用下面的代码，我假设您理解熊猫，如果不理解，您可能想了解它。另外，还不清楚你的2-3秒是在读还是包含了部分数字运算，尽管如此，多重处理会大大加快速度。要点是将所有文件作为数据帧读取，连接它们，然后进行分析

为了对我有用，我在有大量vCPU和内存的现场实例上运行了这个。我发现像c5n这样经过网络优化的实例——例如，机器学习中的n和inf1在读/写方面比T或M实例类型快得多

我的用例是读取2000个“目录”，每个目录中大约有1200个文件，并对它们进行分析。多线程比单线程快几个数量级

文件1：您的主脚本

# create script.py file
import os
from multiprocessing import Pool
from itertools import repeat
import pandas as pd
import json
from utils_file_handling import *

ufh = file_utilities() #instantiate the class functions - see below (second file)

bucket = 'your-bucket'
prefix = 'your-prefix/here/' # if you don't have a prefix pass '' (empty string or function will fail)

#define multiprocessing function - get to know this to use multiple processors to read files simultaneously
def get_dflist_multiprocess(keys_list, num_proc=4):
    with Pool(num_proc) as pool:
        df_list = pool.starmap(ufh.reader_json, zip(repeat(bucket), keys_list), 15)
        pool.close()
        pool.join()
    return df_list

#create your master keys list upfront; you can loop through all or slice the list to test
keys_list = ufh.get_keys_from_prefix(bucket, prefix)
# keys_list = keys_list[0:2000] # as an exampmle

num_proc = os.cpu_count() #tells you how many processors your machine has; function above defaults to 4 unelss given
df_list = get_dflist_multiprocess(keys_list, num_proc=num_proc) #collect dataframes for each file
df_new = pd.concat(df_list, sort=False) 
df_new = df_new.reset_index(drop=True)
# do your analysis on the dataframe 

文件2：类函数

#utils_file_handling.py
# create this in a separate file; name as you wish but change the import in the script.py file
import boto3
import json
import pandas as pd   

#define client and resource
s3sr = boto3.resource('s3')
s3sc = boto3.client('s3')

class file_utilities:
    """file handling function"""

    def get_keys_from_prefix(self, bucket, prefix):
        '''gets list of keys and dates for given bucket and prefix'''
        keys_list = []
        paginator = s3sr.meta.client.get_paginator('list_objects_v2')
        # use Delimiter to limit search to that level of hierarchy
        for page in paginator.paginate(Bucket=bucket, Prefix=prefix, Delimiter='/'):
            keys = [content['Key'] for content in page.get('Contents')]
            print('keys in page: ', len(keys))
            keys_list.extend(keys)
        return keys_list

    def read_json_file_from_s3(self, bucket, key):
        """read json file"""
        bucket_obj = boto3.resource('s3').Bucket(bucket)
        obj = boto3.client('s3').get_object(Bucket=bucket, Key=key)
        data = obj['Body'].read().decode('utf-8')
        return data

    # you may need to tweak this for your ['dict-name'] example; I think I have it correct
    def reader_json(self, bucket, key):
        '''returns dataframe'''
        return pd.DataFrame(json.loads(self.read_json_file_from_s3(bucket, key))['dict-name'])

---

您是否考虑过使用该服务？SQL不太好您还可以使用在parallelraise RuntimeError中运行的S3 select查询RuntimeError：在当前进程完成引导阶段之前，已尝试启动新进程。这可能意味着您没有使用fork启动子进程，并且忘记了在主模块中使用正确的习惯用法：if _uname __=='_umain _;：冻结\u支持。。。如果程序不会被冻结以生成可执行文件，则可以省略冻结支持行。做了一些可能会修正的编辑。否则你可以发布你的代码吗？将import pandas as pd添加到第二个文件中，并将import utils_file_handling import*更改为from utils_file_handling import*