如何在Python中从S3读取Avro文件?
我有一堆Avro文件,我想从S3中逐一读取。我把文件读成字节没有问题,但我想知道在那之后你如何迭代整个文件。当前代码:如何在Python中从S3读取Avro文件?,python,amazon-s3,avro,Python,Amazon S3,Avro,我有一堆Avro文件,我想从S3中逐一读取。我把文件读成字节没有问题,但我想知道在那之后你如何迭代整个文件。当前代码: conn = boto.s3.connect_to_region("us-east-1") my_bucket=boto.s3.bucket.Bucket(conn, "my_bucket") my_key = my_bucket.get_key("folder/file.avro") raw_bytes = my_key.read() test_schema = ''' {
conn = boto.s3.connect_to_region("us-east-1")
my_bucket=boto.s3.bucket.Bucket(conn, "my_bucket")
my_key = my_bucket.get_key("folder/file.avro")
raw_bytes = my_key.read()
test_schema = '''
{
"namespace": "com.company",
"type": "record",
"name": "MimeMessage_v2",
"fields": [
{
"name": "record_timestamp",
"type": "long"
},
{
"name": "contents",
"type": "bytes"
}
],
"message_id": 2
}
'''
schema = avro.schema.Parse(test_schema)
#this is the problematic section
dreader = DatumReader(schema, schema)
v = dreader.read(raw_bytes)
我想知道如何正确读取包含Avro文件字节的变量。以下是在Python 3中对我有效的方法之一:
from avro.datafile import DataFileReader
avro_bytes = io.BytesIO(raw_bytes)
reader = DataFileReader(avro_bytes, avro.io.DatumReader())
for line in reader:
print(line)