av一区二区在线观看_亚洲男人的天堂网站_日韩亚洲视频_在线成人免费_欧美日韩精品免费观看视频_久草视

您的位置:首頁(yè)技術(shù)文章
文章詳情頁(yè)

python - 如何在scrapy中帶cookie訪問(wèn)?

瀏覽:105日期:2022-09-05 15:34:56

問(wèn)題描述

簡(jiǎn)單的通過(guò)scrapy訪問(wèn)雪球都報(bào)錯(cuò),我知道要先訪問(wèn)一次雪球,需要cookie信息才能真正打開(kāi)連接。scrapy據(jù)說(shuō)可以不用在意cookie,會(huì)自動(dòng)獲取cookie。我按照這個(gè)連接在middleware里已經(jīng)啟用cookie,http://stackoverflow.com/ques...,但為什么還是會(huì)返回404錯(cuò)誤?搜索了幾天都沒(méi)找到答案。郁悶啊,求幫忙給個(gè)簡(jiǎn)單代碼如何訪問(wèn),謝謝了

class XueqiuSpider(scrapy.Spider): name = 'xueqiu' start_urls = 'https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1' headers = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8','Accept-Language': 'zh-CN,zh;q=0.8','Connection': 'keep-alive','Host': 'www.zhihu.com','Upgrade-Insecure-Requests': '1','User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36' } def __init__(self, url = None):self.user_url = url def start_requests(self):yield scrapy.Request( url = self.start_urls, headers = self.headers, meta = {’cookiejar’: 1 }, callback = self.request_captcha) def request_captcha(self,response):print response

錯(cuò)誤日志。

2017-03-04 12:42:02 [scrapy.core.engine] INFO: Spider opened2017-03-04 12:42:02 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)2017-03-04 12:42:02 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023********Current UserAgent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6************2017-03-04 12:42:12 [scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <200 https://xueqiu.com/robots.txt>Set-Cookie: aliyungf_tc=AQAAAGFYbBEUVAQAPSHDc8pHhpYZKUem; Path=/; HttpOnly2017-03-04 12:42:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://xueqiu.com/robots.txt> (referer: None)********Current UserAgent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6************2017-03-04 12:42:12 [scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <404 https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1>Set-Cookie: aliyungf_tc=AQAAAPTfyyJNdQUAPSHDc8KmCkY5slST; Path=/; HttpOnly2017-03-04 12:42:12 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1> (referer: None)2017-03-04 12:42:12 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1>: HTTP status code is not handled or not allowed2017-03-04 12:42:12 [scrapy.core.engine] INFO: Closing spider (finished)2017-03-04 12:42:12 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

問(wèn)題解答

回答1:

又試了一下.. 確實(shí)不需要登錄哦.. 是我想多了... 直接先請(qǐng)求一下xueqiu.com,拿到cookie后再請(qǐng)求一下API的地址就可以了.. 原來(lái)如此..

==============羞恥的分割線=============

經(jīng)我驗(yàn)證,你需要登錄...

import scrapyimport hashlibfrom scrapy.http import FormRequest, Requestclass XueqiuScrapeSpider(scrapy.Spider): name = 'xueqiu_scrape' allowed_domains = ['xueqiu.com'] def start_requests(self):m = hashlib.md5()m.update(b'your password') # 在這里填入你的密碼password = m.hexdigest().upper()form_data={ 'telephone': 'your account', # 在這里填入你的用戶名 'password': password, 'remember_me': str(), 'areacode': '86',}print(form_data)return [FormRequest( url='https://xueqiu.com/snowman/login', formdata=form_data, meta={'cookiejar': 1}, callback=self.loged_in )] def loged_in(self, response):# print(response.url)return [Request( url='https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1', meta={'cookiejar': response.meta['cookiejar']}, callback=self.get_result, )] def get_result(self, response):print(response.body)

另外,網(wǎng)站確實(shí)對(duì)User-Agent進(jìn)行了驗(yàn)證,可以在settings.py中進(jìn)行設(shè)置,當(dāng)然自己寫(xiě)在爬蟲(chóng)文件里也可以。密碼是MD5加密后的字符串。哦對(duì),補(bǔ)充一點(diǎn),因?yàn)槲沂怯檬謾C(jī)注冊(cè)的,所以form_data是這些字段,如果你是其他方式,只需要用Chrome工具看一下POST請(qǐng)求有哪些參數(shù),自己修改一下form_data的內(nèi)容就行了。

回答2:

哈哈,謝謝咯,解決了幾天的困惑。之前也通過(guò)request來(lái)做不需要登錄,貼下代碼,

session = requests.Session()session.headers = { ’User-Agent’: ’Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36’}session.get(’https://xueqiu.com’)for page in range(1,100): url = ’https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=%s&size=1’ % page print url r = session.get(url)#print r.json().list a = r.text

標(biāo)簽: Python 編程
相關(guān)文章:
主站蜘蛛池模板: 成年网站在线观看 | 中文字幕国产视频 | 国产在线一区二区三区 | 日韩久草| a黄视频| 午夜视频一区二区三区 | 久久高清| 国产91在线视频 | 中文字幕在线播放不卡 | 日韩中文在线视频 | 日韩av在线播 | 久久久精品国产 | 美女视频三区 | 欧美一级黄色网 | 亚洲欧美日韩精品久久亚洲区 | 亚洲性网 | 欧美性视频在线播放 | 国产一区二区三区视频在线观看 | 久久精品99| 欧美日韩一区在线观看 | 国产精品视频免费观看 | av看看| 69av网| 成人av一区二区三区 | 欧洲成人午夜免费大片 | 97久久精品午夜一区二区 | 欧美日韩三级 | 久久精品黄色 | 亚洲精品在线看 | 久久成人国产 | 亚洲国产成人精品女人久久久野战 | 国产一区二区三区四区区 | 97免费在线观看视频 | 国产激情在线 | 北条麻妃视频在线观看 | h肉视频 | 欧美黑人一区 | 中文字幕精品一区二区三区精品 | 亚洲国产成人久久久 | 国产小视频精品 | 国产av毛片 |