Python 将JSON从RESTAPI转换为Azure Synapse Analytics
我需要将时间序列数据从API消化到Synapse Analytics数据库。API返回JSON,我想将其转换为包含日期、值和时间序列名称的表。我能够用Python返回以下JSON:Python 将JSON从RESTAPI转换为Azure Synapse Analytics,python,azure,azure-data-factory,azure-databricks,azure-synapse,Python,Azure,Azure Data Factory,Azure Databricks,Azure Synapse,我需要将时间序列数据从API消化到Synapse Analytics数据库。API返回JSON,我想将其转换为包含日期、值和时间序列名称的表。我能够用Python返回以下JSON: import requests from pandas.io.json import json_normalize url = 'https://api.citivelocity.com/markets/analytics/chartingbe/rest/external/authed/data?client_id
import requests
from pandas.io.json import json_normalize
url = 'https://api.citivelocity.com/markets/analytics/chartingbe/rest/external/authed/data?client_id='+CitiClientId
payload = {'startDate': 20200705, 'endDate': 20200712, 'tags': ['FX.SPOT.EUR.CHF.CITI'],'frequency':'DAILY'}
headers = {'authorization': 'Bearer'+access_token}
response = requests.post(url, json=payload, headers=headers)
print(response.text)
->
{“频率”:“每日”,“正文”:{“外汇现货.欧元.瑞士法郎.花旗”:{“x”:[202007062000707202007082020070920200710],“c”:[1.06365,1.06255,1.06265,1.06155,1.06325],“类型”:“系列”},“状态”:“正常”}
不幸的是,对于邮递员或Azure数据工厂,我没有得到任何返回的数据。使用ADF和Postman调用API是成功的,但我猜请求中有些地方不正确
ADF将是我的首选,但如果可以将数据直接写入Synapse,则使用databricks是可以的
->如何在python中将JSON转换为数据帧(日期、值和时间序列的名称)?从dataframe,我想我会设法写入Synapse。我可以限制数据量,所以使用json_normalize应该可以吗
一个诱人的选择是先将json保存到数据库,然后用sql进行转换,但我认为用python进行转换将是一个更优雅、更健壮的解决方案
CREATE TABLE #SERIES WITH (DISTRIBUTION=ROUND_ROBIN) AS SELECT 'FX.SPOT.EUR.CHF.CITI' AS SERIES_NAME, '20200706, 20200707, 20200708, 20200709, 20200710' as dt, '1.06365, 1.06255, 1.06265, 1.06155, 1.06325' as cl
DECLARE @DT NVARCHAR(400)
DECLARE @CL NVARCHAR(400)
SELECT @DT=DT FROM #SERIES
SELECT @CL=CL FROM #SERIES
SELECT
DT.SERIES_NAME, DT.VALUE AS DT, CL.VALUE AS CL
FROM
(
SELECT row_number() OVER (ORDER BY (SELECT NULL)) AS RN, SERIES_NAME, LTRIM(dt.value) AS VALUE
FROM #SERIES
CROSS APPLY STRING_SPLIT(@DT, ',') dt
) DT
JOIN
(SELECT row_number() OVER (ORDER BY (SELECT NULL)) AS RN, SERIES_NAME, LTRIM(CL.value) AS VALUE
FROM #SERIES
CROSS APPLY STRING_SPLIT(@CL, ',') CL
) CL ON DT.SERIES_NAME = CL.SERIES_NAME AND DT.RN = CL.RN
->
谢谢
SERIES_NAME DT CL
FX.SPOT.EUR.CHF.CITI 20200706 1.06365
FX.SPOT.EUR.CHF.CITI 20200707 1.06255
FX.SPOT.EUR.CHF.CITI 20200708 1.06265
FX.SPOT.EUR.CHF.CITI 20200709 1.06155
FX.SPOT.EUR.CHF.CITI 20200710 1.06325