Quantcast
Channel: Active questions tagged header - Stack Overflow
Viewing all articles
Browse latest Browse all 646

Not receiving headers Scrapy ProxyMesh

$
0
0

I am quite new to Scrapy / ProxyMesh.My request to the Proxymesh server seems to be working as I see my bandwith consumption on the ProxyMesh website, and the meta.proxy is correct in my logs.However, when I log the response headers in Scrapy, I do not receive the X-Proxymesh-IP that I am supposed to receive.Here is my code. What am I doing wrong?

This is my middleware

class Proxymesh(object): def __init__(self):    logging.debug('Initialized Proxymesh middleware')    self.proxy_ip = 'http://host:port' def process_request(self, request, spider):    logging.debug('Processing request through proxy IP: '+ self.proxy_ip)    request.meta['proxy'] = self.proxy_ip

These are my settings in my spider

custom_settings = {"DOWNLOADER_MIDDLEWARES": {"projectName.middlewares.proxymesh.Proxymesh" : 1,     }

This is what the response headers look like

['Set-Cookie']:['__cfduid=d88d4e4cb7... HttpOnly']['Vary']:['User-Agent,Accept-Encoding']['Server']:['cloudflare-nginx']['Date']:['Thu, 19 Oct 2017 10...38:10 GMT']['Cf-Ray']:['3b031b30cbef1565-CDG']['Content-Type']:['text/html; charset=UTF-8']

Thank you for your help


Viewing all articles
Browse latest Browse all 646

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>