問題描述

import urllib.requestimport reweb=urllib.request.urlopen(’https://www.gpyh.com/pricebuy/index?pageNum=1&hasStock=&goodsStandardId=1931&materialDictCode=&materialGroupCode=037001&diameter=&length=&brandId=&merchantId=’)neirong=web.read()def getPage(self,pageIndex): url = self.siteURL + '?pageNum=' + str(pageIndex) request = urllib2.Request(url) response = urllib2.urlopen(request) return response.read().decode(’gbk’)jiangrenhua=neirong.decode(’UTF-8’)RegularExpression=’<td>(.*)</td>’Valuable=re.findall(RegularExpression,jiangrenhua)information=[]for i in range(173): print(Valuable[i]

問題解答

?pageNum=' + str(pageIndex)

這一個不就是你的頁碼控制嗎？登錄后才看到那就用cookie或者用戶名密碼模擬登錄后獲取

httplib2基本應該是所有http請求的終結者了吧。

import httplib2import urllibhttp = httplib2.Http()url=’要獲取的地址’header={’Accept’:’text/html’, ’Accept-Encoding’:’gzip, deflate, sdch’, ’Accept-Language’:’zh-CN,zh;q=0.8’, ’Cache-Control’:’max-age=0’, ’Connection’:’keep-alive’, ’Cookie’:’cookie內容’, ’Upgrade-Insecure-Requests’:’1’, ’User-Agent’:’Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36’} #要有登陸狀態(tài)才能翻頁就要模擬登陸后把cookie放進去body_value={’username’:’test’,’password’:’123456’} #表單的所有內容body_value=urllib.urlencode(body_value) #utf8編碼response, content = http.request(url, ’GET’, headers=header,body=body_value) #GET或者POST方法response.encoding = ’utf-8’#content就是返回內容