I am quite new to Scrapy / ProxyMesh.My request to the Proxymesh server seems to be working as I see my bandwith consumption on the ProxyMesh website, and the meta.proxy is correct in my logs.However, when I log the response headers in Scrapy, I do not receive the X-Proxymesh-IP that I am supposed to receive.Here is my code. What am I doing wrong?
This is my middleware
class Proxymesh(object): def __init__(self): logging.debug('Initialized Proxymesh middleware') self.proxy_ip = 'http://host:port' def process_request(self, request, spider): logging.debug('Processing request through proxy IP: '+ self.proxy_ip) request.meta['proxy'] = self.proxy_ip
These are my settings in my spider
custom_settings = {"DOWNLOADER_MIDDLEWARES": {"projectName.middlewares.proxymesh.Proxymesh" : 1, }
This is what the response headers look like
['Set-Cookie']:['__cfduid=d88d4e4cb7... HttpOnly']['Vary']:['User-Agent,Accept-Encoding']['Server']:['cloudflare-nginx']['Date']:['Thu, 19 Oct 2017 10...38:10 GMT']['Cf-Ray']:['3b031b30cbef1565-CDG']['Content-Type']:['text/html; charset=UTF-8']
Thank you for your help