[2019年9月4日]原网站反爬措施更新了, 代码更改如下

[zklar]

原网站反爬虫机制做了更改. 代码也需要相应更改(到2019年9月4日为止,更改两个地方即可绕过反爬机制)
①将AoiSolaSpider.py中的allowed_domains = ["www.mm131.com"]变为allowed_domains = ["www.mm131.com", "www.mm131.net"] (这里解决的是content被过滤的问题)
②将middlewares.py下AoisolasSpiderMiddleware类中process_request函数的内容整个换成: request.headers['referer'] = "http://www.mm131.com/?zzaqkey=4087969942"
(绕过防盗链)

[bhtomming]

2020-06-15 15:06:24 [scrapy.core.engine] DEBUG: Crawled (302) <GET https://img1.mmmw.net/pic/2735/1.jpg> (referer: http://www.mm131.com/?zzaqkey=4087969942)
2020-06-15 15:06:24 [scrapy.pipelines.files] WARNING: File (code: 302): Error downloading file from <GET https://img1.mmmw.net/pic/2735/1.jpg> referred in http://www.mm131.com/?zzaqkey=4087969942

更新了还是不可以，出现302跳转

[huimark]

可以吧。request.headers['referer'] = "http://www.mm131.com”

[bhtomming]

嗯，可以了，只是目录的最后一张都会跳出Error

[bhtomming]

图片能正常浏览