python使用post和Cookie获取多个页面

我正在获取一个测试站点。该网站使用POST方法以及cookies。（不确定cookies是否重要，但我倾向于认为它们是……）

该应用程序将显示一个页面，其中包含一个“下一步按钮”以生成后续页面。我已经使用LiveHttpHeaders/Firefof来确定查询中应该包含哪些post数据，以及设置cookie的事实。我还验证了如果浏览器设置了cookie，则该页面不起作用

我正在努力找出我在考试中错过了什么/把事情搞砸了。示例代码显示了我试图获取的第一页和第二页的查询/发布数据

我在网上搜索过，也尝试过许多不同的尝试，所以我很确定我错过了一些简单的东西

如有任何想法/意见，我们将不胜感激

#!/usr/bin/python

#test python script
import re
import urllib
import urllib2
import sys, string, os
from  mechanize import Browser
import mechanize
import cookielib            
########################
#
# Parsing App Information
########################

# datafile

cj = "p"
COOKIEFILE = 'cookies.lwp'
#cookielib = 1

urlopen = urllib2.urlopen
#cj = urllib2.cookielib.LWPCookieJar()       
cj = cookielib.LWPCookieJar()       
#cj = ClientCookie.LWPCookieJar()       
Request = urllib2.Request
br = Browser()

if cj != None:
  print "sss"
#install the CookieJar for the default CookieProcessor
  if os.path.isfile(COOKIEFILE):
      cj.load(COOKIEFILE)
      print "foo\n"
  if cookielib:
      opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
      urllib2.install_opener(opener)
      print "foo2\n"

user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
values1 = {'name' : 'Michael Foord',
          'location' : 'Northampton',
          'language' : 'Python' }
headers = { 'User-Agent' : user_agent }



if __name__ == "__main__":
# main app

  baseurl="https://pisa.ucsc.edu/class_search/index.php"
  print "b = ",baseurl
  print "b = ",headers
  query="action=results&binds%5B%3Aterm%5D=2100&binds%5B%3Areg_status%5D=O&binds%5B%3Asubject%5D=&binds%5B%3Acatalog_nbr_op%5D=%3D&binds%5B%3Acatalog_nbr%5D=&binds%5B%3Atitle%5D=&binds%5B%3Ainstr_name_op%5D=%3D&binds%5B%3Ainstructor%5D=&binds%5B%3Age%5D=&binds%5B%3Acrse_units_op%5D=%3D&binds%5B%3Acrse_units_from%5D=&binds%5B%3Acrse_units_to%5D=&binds%5B%3Acrse_units_exact%5D=&binds%5B%3Adays%5D=&binds%5B%3Atimes%5D=&binds%5B%3Aacad_career%5D="


  request = urllib2.Request(baseurl, query, headers)
  response = urllib2.urlopen(request)

  print "gggg \n"
  #print req
  print "\n gggg 555555\n"

  print "res = ",response
  x1 = response.read()
  #x1 = res.read()
  print x1
  #sys.exit()

  cj.save(COOKIEFILE)    # resave cookies
  if cj is None:
      print "We don't have a cookie library available - sorry."
      print "I can't show you any cookies."
  else:
      print 'These are the cookies we have received so far :'
      for index, cookie in enumerate (cj):
          print index, '  :  ', cookie

  cj.save(COOKIEFILE)  

  print "ffgg \n"
  for index, cookie in enumerate (cj):
       print index, '  :  ', cookie


  #baseurl ="http://students.yale.edu/oci/resultList.jsp"
  baseurl="https://pisa.ucsc.edu/class_search/index.php"

  query="action=next&Rec_Dur=100&sel_col%5Bclass_nbr%5D=1&sel_col%5Bclass_id%5D=1&sel_col%5Bclass_title%5D=1&sel_col%5Btype%5D=1&sel_col%5Bdays%5D=1&sel_col%5Btimes%5D=1&sel_col%5Binstr_name%5D=1&sel_col%5Bstatus%5D=1&sel_col%5Benrl_cap%5D=1&sel_col%5Benrl_tot%5D=1&sel_col%5Bseats_avail%5D=1&sel_col%5Blocation%5D=1"

  request = urllib2.Request(baseurl, query, headers)
  response = urllib2.urlopen(request)

  print "gggg \n"
  #print req
  print "\n gggg 555555\n"

  print "res = ",response
  x1 = response.read()
  #x1 = res.read()
  print x1
  sys.exit()


  req = Request(baseurl, query, headers)
  print "gggg \n"
  #print req
  print "\n gggg 555555\n"
  #br.open(req)

  res = urlopen(req)
  print "gggg 000000000000\n"
  x1 = res.read()
  print x1


  sys.exit()

谢谢你的任何想法/建议

是的。。我知道。。脚本/测试非常糟糕

---

-汤姆

我真的不知道出了什么问题。这可能没有帮助，但有一种不同的方法：

为每个第一页提交打开一个表行，然后在cookie中仅保留rowId，并添加到具有后续访问的行中

---

tf.

怎么了？什么不起作用？是的，这是一个非常糟糕的代码，但是你得到了什么错误？如果你收到页面，怎么了？嗨。。。永远无法使urllib/cookie/referer正常工作。最后切换到使用pycurl来完成页面的数据获取/提取。谢谢大家！