Python MongoDB查询请求将数据分割成块
我正在编写一个python脚本来完成以下步骤 查询MongoDB数据库 分析和聚合结果 通过REST API将数据上载到ServiceNow表 但是,脚本可以工作,数据集太大,其余事务在60秒后超时(连接由目标的ServiceNow服务器关闭) 我需要将数据分割成块,并为每个数据块发送单独的REST事务,以确保通过POST发送完整的数据集,而不会达到超时限制 我如何通过修改下面的脚本来实现这一目标Python MongoDB查询请求将数据分割成块,python,mongodb,rest,servicenow,chunked,Python,Mongodb,Rest,Servicenow,Chunked,我正在编写一个python脚本来完成以下步骤 查询MongoDB数据库 分析和聚合结果 通过REST API将数据上载到ServiceNow表 但是,脚本可以工作,数据集太大,其余事务在60秒后超时(连接由目标的ServiceNow服务器关闭) 我需要将数据分割成块,并为每个数据块发送单独的REST事务,以确保通过POST发送完整的数据集,而不会达到超时限制 我如何通过修改下面的脚本来实现这一目标 #!/usr/bin/env python from config import * impo
#!/usr/bin/env python
from config import *
import os, sys
mypath = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.join(mypath, "api-python-client"))
from apiclient.mongo import *
from pymongo import MongoClient
import json
import requests
from bson.json_util import dumps
client = MongoClient(mongo_uri)
#Create ServiceNow URL
svcnow_url = create_svcnow_url('u_imp_cmps')
#BITSDB Nmap Collection
db = client[mongo_db]
#Aggregate - RDBMS equivalent to Alias select x as y
#Rename fields to match ServiceNow field names
computers = db['computer'].aggregate([
{"$unwind": "$hostnames"},
{"$project" : {
"_id":0,
"u_hostname": "$hostnames.name",
"u_ipv4": "$addresses.ipv4",
"u_status": "$status.state",
"u_updated_timestamp": "$last_seen"
}}
])
j = dumps({"records":computers})
#print(j)
#Set proper headers
headers = {"Content-Type":"application/json","Accept":"application/json"}
#Build HTTP Request
response = requests.post(url=svcnow_url, auth=(svcnow_user, svcnow_pwd), headers=headers ,data=j)
#Check for HTTP codes other than 200
if response.status_code != 200:
print('Status:', response.status_code, 'Headers:', response.headers, 'Response Text', response.text, 'Error Response:',response.json())
exit()
#Decode the JSON response into a dictionary and use the data
print('Status:',response.status_code,'Headers:',response.headers,'Response:',response.json())
更新:我有一个计划,但我不知道如何具体实施
- 将光标设置为每个记录1000条的固定批量大小
- 当批处理已满时,创建JSON输出并通过请求发送数据
- 在循环中:继续抓取新批并将每个批发送到目标,直到到达整个数据集
j=dumps({“records”:computers})
将返回一个列表,因此您可以通过调用j[x]
或迭代for循环轻松地指向单个数据条目。ServiceNow应接受这些条目中的每一项
# Set proper headers (these are always the same, so this
# can be assigned outside of the for loop)
headers = {"Content-Type":"application/json","Accept":"application/json"}
for data_point in j:
#Build HTTP Request (Note we are using data_point instead of j)
response = requests.post(url=svcnow_url, auth=(svcnow_user, svcnow_pwd), headers=headers ,data=data_point)
#Check for HTTP codes other than 200
if response.status_code != 200:
print('Status:', response.status_code, 'Headers:', response.headers, 'Response Text', response.text, 'Error Response:',response.json())
else:
# This is a response of success for a single record
print('Status:',response.status_code,'Headers:',response.headers,'Response:',response.json())
exit()
如果您在MongoDB中有100个新条目,这将对ServiceNow进行100次POST调用。您的ServiceNow实例应该能够处理加载,并且您可以非常轻松地识别未能加载的记录
但是,如果出于任何原因需要压缩呼叫数,我建议将列表拆分为“子列表”,如:
有人对至少从哪里开始有什么建议吗?我卡住了。谢谢。嗨,史蒂夫。谢谢你的回答。我遇到了一个问题,循环逐字符(而不是逐记录)在数据上循环。换句话说,如果我执行“for data_point In j:print(data_point)”,它将逐个返回每个字符,而不是每个记录。你知道我该如何缓解这个问题吗?谢谢啊,我想我知道它为什么这么做。变量“j”的数据类型是字符串而不是列表。“print(type(j))”。我可以通过将光标数据转换为列表并迭代列表来修复它。再次感谢你的帮助。
# Set proper headers (these are always the same, so this
# can be assigned outside of the for loop)
headers = {"Content-Type":"application/json","Accept":"application/json"}
# Each POST will send up to 10 records of data
split_size = 10
# Note the two places where our split_size variable is used
for data_point in [j[x:x+split_size] for x in xrange(0, len(j), split_size)]:
#Build HTTP Request (Note we are using data_point instead of j)
response = requests.post(url=svcnow_url, auth=(svcnow_user, svcnow_pwd), headers=headers ,data=data_point)
#Check for HTTP codes other than 200
if response.status_code != 200:
print('Status:', response.status_code, 'Headers:', response.headers, 'Response Text', response.text, 'Error Response:',response.json())
else:
# This is a response of success for a single record
print('Status:',response.status_code,'Headers:',response.headers,'Response:',response.json())
exit()