有没有更好的方法在R中完成这个XML抓取任务?
我有一些XML看起来像:有没有更好的方法在R中完成这个XML抓取任务?,xml,r,xpath,web-scraping,Xml,R,Xpath,Web Scraping,我有一些XML看起来像: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <array> <dict> <key>
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
<dict>
<key>bbox.NE.lat</key>
<string>-27.45433</string>
<key>bbox.NE.lon</key>
<string>153.01474</string>
<key>bbox.SW.lat</key>
<string>-27.45706</string>
<key>bbox.SW.lon</key>
<string>153.01239</string>
<key>crs</key>
<string>EPSG 4326</string>
<key>found</key>
<string>1</string>
</dict>
<array>
<dict>
<key>bbox</key>
<dict>
<key>bbox.NE.lat</key>
<string>-27.45433</string>
<key>bbox.NE.lon</key>
<string>153.01474</string>
<key>bbox.SW.lat</key>
<string>-27.45706</string>
<key>bbox.SW.lon</key>
<string>153.01239</string>
</dict>
<key>centroid</key>
<dict>
<key>lat</key>
<dict>
<key>lat</key>
<string>-27.45513</string>
<key>lon</key>
<string>153.0137</string>
</dict>
</dict>
<key>id</key>
<string>33037721</string>
<key>properties</key>
<dict>
<key>amenity</key>
<string>university</string>
<key>name</key>
<string>Queensland University of Technology</string>
<key>osm_element</key>
<string>way</string>
<key>osm_id</key>
<string>26303436</string>
</dict>
</dict>
</array>
</array>
</plist>
bbox.NE.lat
-27.45433
bbox.NE.lon
153.01474
bbox.SW.lat
-27.45706
bbox.SW.lon
153.01239
crs
EPSG 4326
建立
1.
口技
bbox.NE.lat
-27.45433
bbox.NE.lon
153.01474
bbox.SW.lat
-27.45706
bbox.SW.lon
153.01239
质心
拉特
拉特
-27.45513
朗
153.0137
身份证件
33037721
性质
便利设施
大学
名称
昆士兰科技大学
osm_元素
方式
osm_id
26303436
出于可再现性的考虑,我使用循环获取XML:
# initialize the vector
queries<-(0)
# loop over coordinates - signups$latlong is just a vector of coordinates
# signups$latlong[1] = "51.5130004883%20-0.1230000034
# the space between lat & long is URL encoded
for(i in 1:length(signups$latlong)){
#self-imposed rate-limiting
Sys.sleep(0.05)
# if query returns an error, write NA and move on, also output to console so I can keep an eye on it
if(class(try(queries[i]<- getURL(paste("http://geocoding.cloudmade.com/[API KEY HERE]/geocoding/v2/find.plist?object_type=university&around=",signups$latlong[i],"&results=5&distance=closest", sep="")), silent=T))=="try-error")
{
queries[i]<-NA
print("NA")
print(i)
}
# just a progress indicator
else(print(i))
}
#初始化向量
修复xml后的查询
使用xpath遍历此过程的方法是:
> plist = xmlParse('data.xml')
> xpathSApply(plist, '/plist/array/dict/dict/string', xmlValue)
[1] "-27.45433" "153.01474"
[3] "-27.45706" "153.01239"
[5] "university" "Queensland University of Technology"
[7] "way" "26303436"
您可以像普通一样对输出进行索引
但是,如果节点具有属性,例如您的XML格式不正确,您确定提供的是真实数据吗?这是一个格式问题。现在应该是正确的了。是的,格式错误的XML肯定是一个格式问题!因此,如果我在一个名为output的对象中有14000个以上的XML块,我可以使用类似于(长度为I(output))university的,这听起来是正确的。还可以看到我在上面添加的另一种方式,它可能更适合plist格式;我将plist=xmlTreeParse(url,useInternalNodes=TRUE)
而不显式地getUrl
,并且lappy
覆盖输出,而不是执行循环。好的,这似乎解决了我的后续问题,如果不是所有XML块都有相同数量的节点,该怎么办。我试试看。Martin,我没有意识到我可以直接在xmlTreeParse中使用URL而不首先获取它。如果您想要积分,请单独提交。
> plist = xmlParse('data.xml')
> xpathSApply(plist, '/plist/array/dict/dict/string', xmlValue)
[1] "-27.45433" "153.01474"
[3] "-27.45706" "153.01239"
[5] "university" "Queensland University of Technology"
[7] "way" "26303436"
> xpathSApply(plist, '/plist/array/dict/dict/string[@type='uniname']', xmlValue)
> sapply(getNodeSet(plist, '//key[text() = "name"]'), function(x) xmlValue(getSibling(x)))
[1] "Queensland University of Technology"