I did this spider that checks the stock of references list (around 3500 references).Now the spider takes around 37 seconds to scrap 400 references. The CPU is around 5%, the network card (1Gbps) around 18%. My internet connection is 300Mbps simetric and it's only connected to this computer. Any idea to improve performance?  Is this a good performance? Maybe ISP router is a bottleneck?import scrapyfrom scrapy.crawler import CrawlerProcessclass Spider(scrapy.Spider):    name = Spider    start_urls = ['URLS']    def __init__(self, references=None, *args, **kwargs):        super(ktmSpider, self).__init__(*args, **kwargs)    def parse(self, response):        return scrapy.FormRequest.from_response(            response,            formdata={'UserName': 'username', 'Password': 'password'},            callback=self.after_login        )    def after_login(self, response):        ref = references.pop()        yield scrapy.Request(url=url + ref, callback=self.parse_stock)    def parse_stock(self, response):        self.f.write(response.selector.xpath('//*[@id=priceDetails]/form/div[2]/text()').extract_first() + ',')        self.f.write(response.selector.xpath('//*[@id=priceDetails]/form/div[8]/div[1]/span/span[2]/text()').extract_first() + ',')        self.f.write(response.selector.xpath('//*[@id=priceDetails]/form/div[8]/div[1]/span/span[1]/i/@style').extract_first() + '\n')        while len(references) > 0:            ref = references.pop()            yield scrapy.Request(url=url + ref, callback=self.parse_stock)f = open(references.txt)references = f.read().splitlines()process = CrawlerProcess({    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',    'LOG_LEVEL': 'INFO',    'AUTOTHROTTLE_ENABLED': 'True',    'AUTOTHROTTLE_START_DELAY': '0.1',    'AUTOTHROTTLE_TARGET_CONCURRENCY': '100'})process.crawl(ktmSpider, references=references, stockFile=file.txt)process.start()
Check the stock of references list
python;performance;scrapy
Auto-throttling extension may cause high download delays. Either turn the extension off to see what would the resulting time be, or limit the maximum delay via AUTOTHROTTLE_MAX_DELAY.Also, you may issue multiple requests from the after_login() method instead of keeping the queue of references:def after_login(self, response):    for ref in references:        yield scrapy.Request(url=url + ref, callback=self.parse_stock)Also, instead of writing to the file from a spider directly, you can use a CSV output pipeline:import csvclass CSVWriterPipeline(object):    def __init__(self):        self.writer = csv.writer(open('file.txt', 'wb'))    def process_item(self, item, spider):        self.writer.writerow([item[field1], item[field2], item[field3])        return itemWhere field1, field2, field3 are your item fields:class MyItem(Item):    field1 = Field()    field2 = Field()    field3 = Field()Which should be set in the parse_stock() callback:def parse_stock(self, response):    item = MyItem()    item[field1] = response.xpath('//*[@id=priceDetails]/form/div[2]/text()').extract_first()    item[field2] = response.xpath('//*[@id=priceDetails]/form/div[8]/div[1]/span/span[2]/text()').extract_first()    item[field3] = response.xpath('//*[@id=priceDetails]/form/div[8]/div[1]/span/span[1]/i/@style').extract_first()    return itemThen, you would need to enable the pipeline:process = CrawlerProcess({    'ITEM_PIPELINES', {         '__main__.CSVWriterPipeline': 100    },    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',    'LOG_LEVEL': 'INFO',    'AUTOTHROTTLE_ENABLED': 'True',    'AUTOTHROTTLE_START_DELAY': '0.1',    'AUTOTHROTTLE_TARGET_CONCURRENCY': '100'})Or course, improve the field and item class names to be more meaningful.There are other things, like using local DNS cache, you may try to speed up your spider:Speed up web scraperAnd, as a running Scrapy from script reference topic, please see this post.