Python 在一个漂亮的汤脚本中抓取多页--得到相同的结果

我正在尝试循环使用Python 2.7中的Beauty Soup解析表的脚本

第一个表解析工作并产生预期的结果。第二个循环产生的结果与第一个循环完全相同。
其他详情：

• 如果我手动使用第二个循环用来解析的url，我会得到 我要刮的预定页面。刷新有点延迟
• 我在其他网站上使用这个循环，循环按预期工作

以下是脚本：

    import urllib2
    import csv
    from bs4 import BeautifulSoup # latest version bs4

    week = raw_input("Which week?")
    week = str(week)
    data = []
    first = "http://fantasy.nfl.com/research/projections#researchProjections=researchProjections%2C%2Fresearch%2Fprojections%253Foffset%253D"
    middle = "%2526position%253DO%2526sort%253DprojectedPts%2526statCategory%253DprojectedStats%2526statSeason%253D2015%2526statType%253DweekProjectedStats%2526statWeek%253D"
    last = "%2Creplace"
    page_num = 1
    for page_num in range(1,3):
        page_mult = (page_num-1) * 25 +1
        next = str(page_mult)
        url = first + next + middle + week + last
    print url #I added this in order to check my output
    html = urllib2.urlopen(url).read()
    soup = BeautifulSoup(html,"lxml")
    table = soup.find('table', attrs={'class':'tableType-player hasGroups'})
    table_body = table.find('tbody')

    rows = table_body.find_all('tr')
    for row in rows:
        cols = row.find_all('td')
        cols = [ele.text.strip() for ele in cols]
        data.append([ele for ele in cols if ele]) # Get rid of empty values
    b = open('NFLtable.csv', 'w')
    a = csv.writer(b)
    a.writerows(data)
    b.close()
    page_num =page_num+1
    print data

---

在实际页面上，他们使用AJAX请求额外的结果，并使用JSON响应和一些HTML作为值之一

我稍微修改了您的代码，请尝试一下：

import urllib2
import urllib
import csv
from bs4 import BeautifulSoup  # latest version bs4
import json

week = raw_input("Which week?")
week = str(week)
data = []
url_format = "http://fantasy.nfl.com/research/projections?offset={offset}&position=O&sort=projectedPts&statCategory=projectedStats&statSeason=2015&statType=weekProjectedStats&statWeek={week}"

for page_num in range(1, 3):
    page_mult = (page_num - 1) * 25 + 1
    next = str(page_mult)
    url = url_format.format(week=week, offset=page_mult)
    print url  # I added this in order to check my output

    request = urllib2.Request(url, headers={'Ajax-Request': 'researchProjections'})
    raw_json = urllib2.urlopen(request).read()
    parsed_json = json.loads(raw_json)
    html = parsed_json['content']

    soup = BeautifulSoup(html, "html.parser")
    table = soup.find('table', attrs={'class': 'tableType-player hasGroups'})
    table_body = table.find('tbody')

    rows = table_body.find_all('tr')
    for row in rows:
        cols = row.find_all('td')
        cols = [ele.text.strip() for ele in cols]
        data.append([ele for ele in cols if ele])  # Get rid of empty values

print data

---

我用week=4进行了测试。

我不确定是否理解，你想刮两页，但用脚本只得到一页？有点。我把第一页刮得很好。当它在循环的第二次迭代中时，它会正确地创建第二个url，但会再次从第一个页面返回结果。因此，如果每页有25条记录，我的CSV中有50条记录，但前25条与后25条相同。如果我在第二个循环之后重复这个，我会再次得到前25条记录。非常感谢你的帮助——这很有效。我不知道刮取和打电话以确保刮取正确数据的区别。