Python Scrapy:从HTML脚本中提取JSON
我试图从HTML脚本中提取(看起来是)JSON数据。网站上的HTML脚本如下所示:Python Scrapy:从HTML脚本中提取JSON,python,html,json,xpath,scrapy,Python,Html,Json,Xpath,Scrapy,我试图从HTML脚本中提取(看起来是)JSON数据。网站上的HTML脚本如下所示: <script> $(document).ready(function(){ var terms = new Verba.Compare.Collections.Terms([{"id":"6436","name":"SUMMER 16","inquiry":true,"ordering":true},{"id":"6517","name":"FALL 16","inquiry":t
<script>
$(document).ready(function(){
var terms = new Verba.Compare.Collections.Terms([{"id":"6436","name":"SUMMER 16","inquiry":true,"ordering":true},{"id":"6517","name":"FALL 16","inquiry":true,"ordering":true}]);
var view = new Verba.Compare.Views.CourseSelector({el: "body", terms: terms});
});
</script>
使用下面的代码,我能够得到完整的脚本
def parse(self, response):
print response.xpath('/html/body/script[2]').extract()
是否有一种简单的方法可以从该脚本中提取“id”、“name”等值。或者,有没有更直接的方法修改xpath?我似乎无法使用firebug对xpath进行更深入的研究。您无法“深入”,因为该元素的内容只是文本。从JavaScript读取JSON并不难:
line = javascript.strip().splitlines()[1]
the_json = line.split('(', 1)[1].split(')', 1)[0]
我将使用正则表达式提取它,类似于:
response.xpath('/html/body/script[2]').re_first('\((\[.*\])\)')
你可以用这个
为了举例说明,首先,让我们使用示例HTML创建一个Scrapy选择器,并获取JavaScript语句:
>>> import scrapy
>>> sample = '''<script>
... $(document).ready(function(){
... var terms = new Verba.Compare.Collections.Terms([{"id":"6436","name":"SUMMER 16","inquiry":true,"ordering":true},{"id":"6517","name":"FALL 16","inquiry":true,"ordering":true}]);
... var view = new Verba.Compare.Views.CourseSelector({el: "body", terms: terms});
... });
... </script>'''
>>> selector = scrapy.Selector(text=sample, type='html')
>>> selector.xpath('//script//text()').extract_first()
u'\n $(document).ready(function(){\n var terms = new Verba.Compare.Collections.Terms([{"id":"6436","name":"SUMMER 16","inquiry":true,"ordering":true},{"id":"6517","name":"FALL 16","inquiry":true,"ordering":true}]);\n var view = new Verba.Compare.Views.CourseSelector({el: "body", terms: terms});\n });\n'
最后,现在您有了
元素,您可以将它传递到js2xml.jsonlike.make_dict()
以获得一个好的Python对象来使用(make_dict
有点命名错误):
注意:您还可以使用快捷方式js2xml.jsonlike.getall()
获取所有看起来像Python dict或list的内容(您得到两个列表,您对第一个列表感兴趣):
提供将JavaScript对象解析为dict的API
例如,如果JavaScript代码包含var data={field:“value”,secondField:“second value”}
;您可以按如下方式提取该数据:
import chompjs
javascript = response.css('script::text').get()
data = chompjs.parse_js_object(javascript)
最终结果是
{'field':'value','secondField':'secondvalue'}
a这会有帮助吗:它有效!经过更多的研究,我找到了一个类似于你刚才发布的方法。不过,你的要干净得多。谢谢,非常欢迎。如果它解决了您的问题,请随时将此标记为答案。
>>> import js2xml
>>> jssnippet = selector.xpath('//script//text()').extract_first()
>>> jstree = js2xml.parse(jssnippet)
>>> jstree
<Element program at 0x7fc7c6bae1b8>
>>> print(js2xml.pretty_print(jstree))
<program>
<functioncall>
<function>
<dotaccessor>
<object>
<functioncall>
<function>
<identifier name="$"/>
</function>
<arguments>
<identifier name="document"/>
</arguments>
</functioncall>
</object>
<property>
<identifier name="ready"/>
</property>
</dotaccessor>
</function>
<arguments>
<funcexpr>
<identifier/>
<parameters/>
<body>
<var name="terms">
<new>
<dotaccessor>
<object>
<dotaccessor>
<object>
<dotaccessor>
<object>
<identifier name="Verba"/>
</object>
<property>
<identifier name="Compare"/>
</property>
</dotaccessor>
</object>
<property>
<identifier name="Collections"/>
</property>
</dotaccessor>
</object>
<property>
<identifier name="Terms"/>
</property>
</dotaccessor>
<arguments>
<array>
<object>
<property name="id">
<string>6436</string>
</property>
<property name="name">
<string>SUMMER 16</string>
</property>
<property name="inquiry">
<boolean>true</boolean>
</property>
<property name="ordering">
<boolean>true</boolean>
</property>
</object>
<object>
<property name="id">
<string>6517</string>
</property>
<property name="name">
<string>FALL 16</string>
</property>
<property name="inquiry">
<boolean>true</boolean>
</property>
<property name="ordering">
<boolean>true</boolean>
</property>
</object>
</array>
</arguments>
</new>
</var>
<var name="view">
<new>
<dotaccessor>
<object>
<dotaccessor>
<object>
<dotaccessor>
<object>
<identifier name="Verba"/>
</object>
<property>
<identifier name="Compare"/>
</property>
</dotaccessor>
</object>
<property>
<identifier name="Views"/>
</property>
</dotaccessor>
</object>
<property>
<identifier name="CourseSelector"/>
</property>
</dotaccessor>
<arguments>
<object>
<property name="el">
<string>body</string>
</property>
<property name="terms">
<identifier name="terms"/>
</property>
</object>
</arguments>
</new>
</var>
</body>
</funcexpr>
</arguments>
</functioncall>
</program>
>>> jstree.xpath('//var[@name="terms"]')
[<Element var at 0x7fc7c565e638>]
>>> jstree.xpath('//var[@name="terms"]/new/arguments/*')
[<Element array at 0x7fc7c565e5a8>]
>>> jstree.xpath('//var[@name="terms"]/new/arguments/*')[0]
<Element array at 0x7fc7c565e5a8>
>>> js2xml.jsonlike.make_dict(jstree.xpath('//var[@name="terms"]/new/arguments/*')[0])
[{'ordering': True, 'inquiry': True, 'id': '6436', 'name': 'SUMMER 16'}, {'ordering': True, 'inquiry': True, 'id': '6517', 'name': 'FALL 16'}]
>>>
>>> js2xml.jsonlike.getall(jstree)
[[{'ordering': True, 'inquiry': True, 'id': '6436', 'name': 'SUMMER 16'}, {'ordering': True, 'inquiry': True, 'id': '6517', 'name': 'FALL 16'}], {'el': 'body', 'terms': 'terms'}]
import chompjs
javascript = response.css('script::text').get()
data = chompjs.parse_js_object(javascript)